← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #87907

[Bug 1954785] [NEW] Session semantic violated when sending FloatingIP disassociation update during port deletion

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

In plugins.ml2.plugin.Ml2Plugin.delete_port we are calling L3RouterPlugin.disassociate_floatingips (db.l3_db.L3_NAT_dbonly_mixin.disassociate_floatingips) which at the end calls neutron_lib.callbacks.registry.publish to send FLOATING_IP AFTER_UPDATE notification.
Because call to disassociate_floatingips is performed within context of DB transaction in delete_port,
 plugins.ml2.ovo_rpc._ObjectChangeHandler._is_session_semantic_violated check in plugins.ml2.ovo_rpc._ObjectChangeHandler.handle_event listening to all AFTER_ notifications, will log a following warning and PREVENT notification from being sent at all.

WARNING neutron.plugins.ml2.ovo_rpc [req-6588b226-81b6-463d-a376-45488162671f req-3783a80a-16f1-468e-ae74-f9a9b8db8eca tempest-ServersWithSpecificFlavorTestJSON-1266243365 tempest-ServersWithSpecificFlavorTestJSON-1266243365] This handler is supposed to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending resource event: floatingip, after_update. Location:
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 219, in main
    result = function(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 789, in process_request
    proto.__init__(conn_state, self)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 343, in __init__
    self.handle()
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 376, in handle
    self.handle_one_request()
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 450, in handle_one_request
    self.handle_one_response()
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 547, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/paste/urlmap.py", line 216, in __call__
    return app(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/request_id.py", line 58, in __call__
    response = req.get_response(self.application)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
    response = req.get_response(self.application)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
    return request.get_response(self.application)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/keystonemiddleware/keystonemiddleware/auth_token/__init__.py", line 335, in __call__
    response = req.get_response(self._app)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
    application, catch_exc_info=False)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
    response = self.app(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
    return resp(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
    return f(*args, **kwargs)
  File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 115, in wrapped
    f(*args, **kwargs)
  File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 81, in delete
    return self.plugin_deleter(*deleter_args)
  File "/opt/stack/neutron/neutron/common/utils.py", line 668, in inner
    return f(self, context, *args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
    return method(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1809, in delete_port
    context, id, do_notify=False)
  File "/opt/stack/neutron/neutron/db/l3_db.py", line 2115, in disassociate_floatingips
    context, port_id, do_notify)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
    return method(*args, **kwargs)
  File "/opt/stack/neutron/neutron/db/l3_db.py", line 1702, in disassociate_floatingips
    metadata={'association_event': False}))
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish
    _get_callback_manager().publish(resource, event, trigger, payload=payload)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish
    return self.notify(resource, event, trigger, payload=payload)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 102, in _wrapped
    return function(*args, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 167, in notify
    errors = self._notify_loop(resource, event, trigger, **kwargs)
  File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
    callback(resource, event, trigger, **kwargs)
  File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 95, in handle_payload_event
    payload.context, resource, event):
  File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 83, in _is_session_semantic_violated
    stack = traceback.extract_stack()

** Affects: neutron
     Importance: Undecided
     Assignee: Szymon Wróblewski (bluex)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Szymon Wróblewski (bluex)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1954785

Title:
  Session semantic violated when sending FloatingIP disassociation
  update during port deletion

Status in neutron:
  New

Bug description:
  In plugins.ml2.plugin.Ml2Plugin.delete_port we are calling L3RouterPlugin.disassociate_floatingips (db.l3_db.L3_NAT_dbonly_mixin.disassociate_floatingips) which at the end calls neutron_lib.callbacks.registry.publish to send FLOATING_IP AFTER_UPDATE notification.
  Because call to disassociate_floatingips is performed within context of DB transaction in delete_port,
   plugins.ml2.ovo_rpc._ObjectChangeHandler._is_session_semantic_violated check in plugins.ml2.ovo_rpc._ObjectChangeHandler.handle_event listening to all AFTER_ notifications, will log a following warning and PREVENT notification from being sent at all.

  WARNING neutron.plugins.ml2.ovo_rpc [req-6588b226-81b6-463d-a376-45488162671f req-3783a80a-16f1-468e-ae74-f9a9b8db8eca tempest-ServersWithSpecificFlavorTestJSON-1266243365 tempest-ServersWithSpecificFlavorTestJSON-1266243365] This handler is supposed to handle AFTER events, as in 'AFTER it's committed', not BEFORE. Offending resource event: floatingip, after_update. Location:
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/greenthread.py", line 219, in main
      result = function(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 789, in process_request
      proto.__init__(conn_state, self)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 343, in __init__
      self.handle()
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 376, in handle
      self.handle_one_request()
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 450, in handle_one_request
      self.handle_one_response()
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/eventlet/wsgi.py", line 547, in handle_one_response
      result = self.application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/paste/urlmap.py", line 216, in __call__
      return app(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/base.py", line 131, in __call__
      response = req.get_response(self.application)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/base.py", line 131, in __call__
      response = req.get_response(self.application)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/request_id.py", line 58, in __call__
      response = req.get_response(self.application)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
      response = req.get_response(self.application)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
      return request.get_response(self.application)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
      resp = self.call_func(req, *args, **kw)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/keystonemiddleware/keystonemiddleware/auth_token/__init__.py", line 335, in __call__
      response = req.get_response(self._app)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
      application, catch_exc_info=False)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
      app_iter = application(self.environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
      return resp(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
      return resp(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
      response = self.app(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
      return resp(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
      return self.application(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
      return super(Pecan, self).__call__(environ, start_response)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 683, in __call__
      self.invoke_controller(controller, args, kwargs, state)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/pecan/core.py", line 574, in invoke_controller
      result = controller(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
      return f(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
      return f(*dup_args, **dup_kwargs)
    File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
      return f(*args, **kwargs)
    File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 115, in wrapped
      f(*args, **kwargs)
    File "/opt/stack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 81, in delete
      return self.plugin_deleter(*deleter_args)
    File "/opt/stack/neutron/neutron/common/utils.py", line 668, in inner
      return f(self, context, *args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
      return method(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
      return f(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
      return f(*dup_args, **dup_kwargs)
    File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1809, in delete_port
      context, id, do_notify=False)
    File "/opt/stack/neutron/neutron/db/l3_db.py", line 2115, in disassociate_floatingips
      context, port_id, do_notify)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
      return method(*args, **kwargs)
    File "/opt/stack/neutron/neutron/db/l3_db.py", line 1702, in disassociate_floatingips
      metadata={'association_event': False}))
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish
      _get_callback_manager().publish(resource, event, trigger, payload=payload)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish
      return self.notify(resource, event, trigger, payload=payload)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 102, in _wrapped
      return function(*args, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 167, in notify
      errors = self._notify_loop(resource, event, trigger, **kwargs)
    File "/opt/stack/neutron.venv/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
      callback(resource, event, trigger, **kwargs)
    File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 95, in handle_payload_event
      payload.context, resource, event):
    File "/opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py", line 83, in _is_session_semantic_violated
      stack = traceback.extract_stack()

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1954785/+subscriptions
  Thread Previous • Date Previous • • Thread Next