yahoo-eng-team team mailing list archive

[Sebastian Lohff <2114919@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

dhcp_ready_on_ports() might fail during provisioning in
provisioning_complete(), as this method has a transaction guard:
RuntimeError: Must not be called in a transaction

This seems to only happen when multiple ports are provisioned at once
and at least one of the ports (which is not the last port) is a router
port. Tracing this through our infra, this seems to come from the logapi
in neutron-fwaas (which we use in our infrastructure). If a router port
is updated, its AFTER_UPDATE hooks are called. For router ports neutron-
fwaas runs the hook
neutron_fwaas.services.logapi.common.port_callback.NeutronPortCallBack.handle_event,
where log_db_api.get_logs_for_port() calls
fw_plugin_db.get_fwg_attached_to_port(). This is done outside of a
transaction (AFTER_UPDATE doesn't open one).

Adding the respective CONTEXT_READER / CONTEXT_WRITER to
FirewallPluginDb seems to fix the problem. While on it I'd suggest we
also add the annotation to the remaining methods that don't yet have
one.

Tested with Neutron Caracal. I'll propose a patch for this.

Full stack trace:
RuntimeError: Must not be called in a transaction
  File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)
  File "oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)
  File "neutron_lib/db/api.py", line 137, in wrapped
    with excutils.save_and_reraise_exception():
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
  File "oslo_db/api.py", line 144, in wrapper
    with excutils.save_and_reraise_exception() as ectxt:
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
  File "neutron_lib/db/api.py", line 183, in wrapped
    with excutils.save_and_reraise_exception():
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 181, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "neutron/api/rpc/handlers/dhcp_rpc.py", line 532, in dhcp_ready_on_ports
    provisioning_blocks.provisioning_complete(
  File "neutron_lib/db/api.py", line 221, in wrapped
    return f(*args, **kwargs)
  File "neutron/db/provisioning_blocks.py", line 128, in provisioning_complete
    raise RuntimeError(_("Must not be called in a transaction"))

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2114919

Title:
  dhcp_ready_on_ports() + fwaas might fail due to open transaction

Status in neutron:
  New

Bug description:
  dhcp_ready_on_ports() might fail during provisioning in
  provisioning_complete(), as this method has a transaction guard:
  RuntimeError: Must not be called in a transaction

  This seems to only happen when multiple ports are provisioned at once
  and at least one of the ports (which is not the last port) is a router
  port. Tracing this through our infra, this seems to come from the
  logapi in neutron-fwaas (which we use in our infrastructure). If a
  router port is updated, its AFTER_UPDATE hooks are called. For router
  ports neutron-fwaas runs the hook
  neutron_fwaas.services.logapi.common.port_callback.NeutronPortCallBack.handle_event,
  where log_db_api.get_logs_for_port() calls
  fw_plugin_db.get_fwg_attached_to_port(). This is done outside of a
  transaction (AFTER_UPDATE doesn't open one).

  Adding the respective CONTEXT_READER / CONTEXT_WRITER to
  FirewallPluginDb seems to fix the problem. While on it I'd suggest we
  also add the annotation to the remaining methods that don't yet have
  one.

  Tested with Neutron Caracal. I'll propose a patch for this.

  Full stack trace:
  RuntimeError: Must not be called in a transaction
    File "oslo_messaging/rpc/server.py", line 165, in _process_incoming
      res = self.dispatcher.dispatch(message)
    File "oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)
    File "oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      result = func(ctxt, **new_args)
    File "neutron_lib/db/api.py", line 137, in wrapped
      with excutils.save_and_reraise_exception():
    File "oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "neutron_lib/db/api.py", line 135, in wrapped
      return f(*args, **kwargs)
    File "oslo_db/api.py", line 144, in wrapper
      with excutils.save_and_reraise_exception() as ectxt:
    File "oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "oslo_db/api.py", line 142, in wrapper
      return f(*args, **kwargs)
    File "neutron_lib/db/api.py", line 183, in wrapped
      with excutils.save_and_reraise_exception():
    File "oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "neutron_lib/db/api.py", line 181, in wrapped
      return f(*dup_args, **dup_kwargs)
    File "neutron/api/rpc/handlers/dhcp_rpc.py", line 532, in dhcp_ready_on_ports
      provisioning_blocks.provisioning_complete(
    File "neutron_lib/db/api.py", line 221, in wrapped
      return f(*args, **kwargs)
    File "neutron/db/provisioning_blocks.py", line 128, in provisioning_complete
      raise RuntimeError(_("Must not be called in a transaction"))

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2114919/+subscriptions