yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #78356
[Bug 1828375] [NEW] Bulk creation of subports fails with StaleDataError
Public bug reported:
ENV:
Neutron Ocata (10.0.4)
Kola Ansible
Rhel 7.4
One compute node (openvswitch) with 3 virtual machine kubernetes minions with Nested Kuryr.
REPRODUCTION:
Creation of multiple containers in very short time (in fact creation of many subports in very short time)
Some OpenStack ports(trunk subports) will never transition to Active status.
EXCEPTION:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/neutron/services/trunk/rpc/server.py", line 110, in update_trunk_status
trunk.update(status=status)
File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
res = func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/neutron/objects/trunk.py", line 127, in update
super(Trunk, self).update()
File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 618, in update
self._get_composite_keys()))
File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 80, in update_object
db_obj.save(session=context.session)
File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
session.flush()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
self._flush(objects)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
flush_context.execute()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
rec.execute(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
uow
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
mapper, table, update)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
(table.description, len(records), rows))
StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
ADDITIONAL INVESTIGATION:
Actually there are multiple concurrent update calls to database for trunk ports revision_number. Trunks in database:
MariaDB [neutron]> select * from trunks;
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
| admin_state_up | project_id | id | name | port_id | status | standard_attr_id |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
| 1 | 9e8f068a23914209a839be451fe7533a | ce3c9271-70e9-4e4a-bdaf-bcb1a8d3901f | owc4-trunk-1 | e4392b5f-ecc7-4b94-bfe3-a2a8bc001057 | ACTIVE | 56 |
| 1 | 9e8f068a23914209a839be451fe7533a | d9242ed7-4274-4537-962e-5e40b6540cc5 | owc4-trunk-2 | fe0c1041-4ccd-4fd7-b2b9-0c78c6b8c5fa | ACTIVE | 58 |
| 1 | 9e8f068a23914209a839be451fe7533a | f50f1f1e-539b-4915-b410-d4b349c70d4b | owc4-trunk-0 | e4ff627d-37ca-447e-85cc-23347a9ba871 | ACTIVE | 49 |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
3 rows in set (0.00 sec)
standardattributes for trunks:
MariaDB [neutron]> select * from standardattributes where resource_type='trunks';
+----+---------------+---------------------+---------------------+-------------+-----------------+
| id | resource_type | created_at | updated_at | description | revision_number |
+----+---------------+---------------------+---------------------+-------------+-----------------+
| 49 | trunks | 2019-04-16 08:14:10 | 2019-04-25 08:37:32 | | 710 |
| 56 | trunks | 2019-04-16 08:14:45 | 2019-04-25 08:37:32 | | 739 |
| 58 | trunks | 2019-04-16 08:15:09 | 2019-04-25 08:37:56 | | 654 |
+----+---------------+---------------------+---------------------+-------------+-----------------+
3 rows in set (0.00 sec)
MariaDB [neutron]>
database queries dump:
726462 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:36' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728730 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
726445 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727770 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727468 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727550 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:39' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727048 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728694 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728697 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727765 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:41' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727766 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727470 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
728726 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727550 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
726430 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728693 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
728688 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728692 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727770 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:45' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
727051 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:46' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727469 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728732 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727767 Query UPDATE standardattributes SET revision_number=833, updated_at='2019-04-25 11:00:49' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 832
I tried also to queue requests of port updates in agent/server code. After creating locks on methods:
1. def update_subport_bindings(self, context, subports): in file neutron/services/trunk/rpc/server.py
2. def update_trunk_status(self, context, trunk_id, status): in file neutron/services/trunk/rpc/server.py
3. def add_subports(self, context, trunk_id, subports): in file neutron/services/trunk/plugin.py
4. def handle_subports(self, context, resource_type, subports, event_type): in file neutron/services/trunk/drivers/openvswitch/agent/driver.py
Those modification made subport creation work in more stable way.
Interesting fact is that restarting of agent is bringing all ports up.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1828375
Title:
Bulk creation of subports fails with StaleDataError
Status in neutron:
New
Bug description:
ENV:
Neutron Ocata (10.0.4)
Kola Ansible
Rhel 7.4
One compute node (openvswitch) with 3 virtual machine kubernetes minions with Nested Kuryr.
REPRODUCTION:
Creation of multiple containers in very short time (in fact creation of many subports in very short time)
Some OpenStack ports(trunk subports) will never transition to Active status.
EXCEPTION:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
res = self.dispatcher.dispatch(message)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/neutron/services/trunk/rpc/server.py", line 110, in update_trunk_status
trunk.update(status=status)
File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 203, in decorator
res = func(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/neutron/objects/trunk.py", line 127, in update
super(Trunk, self).update()
File "/usr/lib/python2.7/site-packages/neutron/objects/base.py", line 618, in update
self._get_composite_keys()))
File "/usr/lib/python2.7/site-packages/neutron/objects/db/api.py", line 80, in update_object
db_obj.save(session=context.session)
File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
session.flush()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
self._flush(objects)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
flush_context.execute()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
rec.execute(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
uow
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
mapper, table, update)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 728, in _emit_update_statements
(table.description, len(records), rows))
StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
ADDITIONAL INVESTIGATION:
Actually there are multiple concurrent update calls to database for trunk ports revision_number. Trunks in database:
MariaDB [neutron]> select * from trunks;
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
| admin_state_up | project_id | id | name | port_id | status | standard_attr_id |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
| 1 | 9e8f068a23914209a839be451fe7533a | ce3c9271-70e9-4e4a-bdaf-bcb1a8d3901f | owc4-trunk-1 | e4392b5f-ecc7-4b94-bfe3-a2a8bc001057 | ACTIVE | 56 |
| 1 | 9e8f068a23914209a839be451fe7533a | d9242ed7-4274-4537-962e-5e40b6540cc5 | owc4-trunk-2 | fe0c1041-4ccd-4fd7-b2b9-0c78c6b8c5fa | ACTIVE | 58 |
| 1 | 9e8f068a23914209a839be451fe7533a | f50f1f1e-539b-4915-b410-d4b349c70d4b | owc4-trunk-0 | e4ff627d-37ca-447e-85cc-23347a9ba871 | ACTIVE | 49 |
+----------------+----------------------------------+--------------------------------------+--------------+--------------------------------------+----------+------------------+
3 rows in set (0.00 sec)
standardattributes for trunks:
MariaDB [neutron]> select * from standardattributes where resource_type='trunks';
+----+---------------+---------------------+---------------------+-------------+-----------------+
| id | resource_type | created_at | updated_at | description | revision_number |
+----+---------------+---------------------+---------------------+-------------+-----------------+
| 49 | trunks | 2019-04-16 08:14:10 | 2019-04-25 08:37:32 | | 710 |
| 56 | trunks | 2019-04-16 08:14:45 | 2019-04-25 08:37:32 | | 739 |
| 58 | trunks | 2019-04-16 08:15:09 | 2019-04-25 08:37:56 | | 654 |
+----+---------------+---------------------+---------------------+-------------+-----------------+
3 rows in set (0.00 sec)
MariaDB [neutron]>
database queries dump:
726462 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:36' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728730 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
726445 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:37' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727770 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727468 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:38' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727550 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:39' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727048 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728694 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
728697 Query UPDATE standardattributes SET revision_number=830, updated_at='2019-04-25 11:00:40' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 829
727765 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:41' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727766 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727470 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
728726 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:43' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727550 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
726430 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728693 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
728688 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728692 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:44' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727770 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:45' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
727051 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:46' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727469 Query UPDATE standardattributes SET revision_number=832, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 831
728732 Query UPDATE standardattributes SET revision_number=831, updated_at='2019-04-25 11:00:47' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 830
727767 Query UPDATE standardattributes SET revision_number=833, updated_at='2019-04-25 11:00:49' WHERE standardattributes.id = 49 AND standardattributes.revision_number = 832
I tried also to queue requests of port updates in agent/server code. After creating locks on methods:
1. def update_subport_bindings(self, context, subports): in file neutron/services/trunk/rpc/server.py
2. def update_trunk_status(self, context, trunk_id, status): in file neutron/services/trunk/rpc/server.py
3. def add_subports(self, context, trunk_id, subports): in file neutron/services/trunk/plugin.py
4. def handle_subports(self, context, resource_type, subports, event_type): in file neutron/services/trunk/drivers/openvswitch/agent/driver.py
Those modification made subport creation work in more stable way.
Interesting fact is that restarting of agent is bringing all ports up.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1828375/+subscriptions
Follow ups
