yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1862851] [NEW] update_floatingip_statuses: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Arjun Baindur <xagent@xxxxxxxxx>
Date: Tue, 11 Feb 2020 22:45:52 -0000
Reply-to: Bug 1862851 <1862851@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Running Rocky, in a DVR environment

We are seeing repeated errors like this in neutron-server logs. Not sure
at this point what the effect is, or harmless. This is in a build test
environment where a lot of automated VMs get spawned and deleted then
associated/disassociated with existing floating IPs, as well as a lot of
VMs and floating IPs that are created and deleted manually by user. The
logs aren't indicating what the floating IP is, what host, or what
port/VM they were bound to: http://paste.openstack.org/show/789444/

2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api [req-3a00ad99-d6ed-46cc-9f18-6835854799c8 - - - - -] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api Traceback (most recent call last):
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return f(*args, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self.force_reraise()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return f(*dup_args, **dup_kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 273, in update_floatingip_statuses
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     status)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return method(*args, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1524, in update_floatingip_status
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     context, {'status': status}, id=floatingip_id)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 669, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     context, values, validate_filters=False, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 304, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     cls._update_objects(obj, values)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 296, in _update_objects
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     obj.update()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 342, in decorator
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return func(self, *args, **kwargs)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 876, in update
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self._get_composite_keys()))
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/db/api.py", line 82, in update_object
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     db_obj.save(session=context.session)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     session.flush()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self._flush(objects)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     transaction.rollback(_capture_exception=True)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     compat.reraise(exc_type, exc_value, exc_tb)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     flush_context.execute()
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     rec.execute(self)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     uow
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     mapper, table, update)
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 791, in _emit_update_statements
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     (table.description, len(records), rows))
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server [req-3a00ad99-d6ed-46cc-9f18-6835854799c8 - - - - -] Exception during message handling: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 273, in update_floatingip_statuses
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     status)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1524, in update_floatingip_status
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     context, {'status': status}, id=floatingip_id)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 669, in update_object
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     context, values, validate_filters=False, **kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 304, in update_object
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     cls._update_objects(obj, values)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 296, in _update_objects
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     obj.update()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 342, in decorator
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 876, in update
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self._get_composite_keys()))
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/db/api.py", line 82, in update_object
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     db_obj.save(session=context.session)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     session.flush()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self._flush(objects)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     transaction.rollback(_capture_exception=True)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     flush_context.execute()
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     rec.execute(self)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     uow
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     mapper, table, update)
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 791, in _emit_update_statements
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     (table.description, len(records), rows))
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1862851

Title:
  update_floatingip_statuses: StaleDataError: UPDATE statement on table
  'standardattributes' expected to update 1 row(s); 0 were matched.

Status in neutron:
  New

Bug description:
  Running Rocky, in a DVR environment

  We are seeing repeated errors like this in neutron-server logs. Not
  sure at this point what the effect is, or harmless. This is in a build
  test environment where a lot of automated VMs get spawned and deleted
  then associated/disassociated with existing floating IPs, as well as a
  lot of VMs and floating IPs that are created and deleted manually by
  user. The logs aren't indicating what the floating IP is, what host,
  or what port/VM they were bound to:
  http://paste.openstack.org/show/789444/

  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api [req-3a00ad99-d6ed-46cc-9f18-6835854799c8 - - - - -] DB exceeded retry limit.: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api Traceback (most recent call last):
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return f(*args, **kwargs)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self.force_reraise()
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     six.reraise(self.type_, self.value, self.tb)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return f(*dup_args, **dup_kwargs)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 273, in update_floatingip_statuses
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     status)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return method(*args, **kwargs)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1524, in update_floatingip_status
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     context, {'status': status}, id=floatingip_id)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 669, in update_object
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     context, values, validate_filters=False, **kwargs)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 304, in update_object
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     cls._update_objects(obj, values)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 296, in _update_objects
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     obj.update()
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 342, in decorator
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     return func(self, *args, **kwargs)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 876, in update
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self._get_composite_keys()))
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/db/api.py", line 82, in update_object
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     db_obj.save(session=context.session)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     session.flush()
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     self._flush(objects)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     transaction.rollback(_capture_exception=True)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     compat.reraise(exc_type, exc_value, exc_tb)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     flush_context.execute()
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     rec.execute(self)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     uow
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     mapper, table, update)
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 791, in _emit_update_statements
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api     (table.description, len(records), rows))
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  2020-02-11 19:38:52,839.839 30190 ERROR oslo_db.api
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server [req-3a00ad99-d6ed-46cc-9f18-6835854799c8 - - - - -] Exception during message handling: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 273, in update_floatingip_statuses
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     status)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return method(*args, **kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1524, in update_floatingip_status
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     context, {'status': status}, id=floatingip_id)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 669, in update_object
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     context, values, validate_filters=False, **kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 304, in update_object
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     cls._update_objects(obj, values)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 296, in _update_objects
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     obj.update()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 342, in decorator
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/base.py", line 876, in update
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self._get_composite_keys()))
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/neutron/objects/db/api.py", line 82, in update_object
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     db_obj.save(session=context.session)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     session.flush()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     self._flush(objects)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     transaction.rollback(_capture_exception=True)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     flush_context.execute()
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     rec.execute(self)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     uow
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     mapper, table, update)
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server   File "/opt/pf9/neutron/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 791, in _emit_update_statements
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server     (table.description, len(records), rows))
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched.
  2020-02-11 19:38:52,841.841 30190 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1862851/+subscriptions