yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1705351] Re: agent notifier getting amqp NotFound exceptions propagated up to it

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1705351@xxxxxxxxxxxxxxxxxx>
Date: Tue, 22 Aug 2017 16:44:31 -0000
Reply-to: Bug 1705351 <1705351@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/486706
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Submitter: Jenkins
Branch:    master

commit f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Author: Kenneth Giusti <kgiusti@xxxxxxxxx>
Date:   Mon Jul 24 13:36:17 2017 -0400

    Prevent rabbit from raising unexpected exceptions
    
    Publishing a message using the kombu connection autoretry method may
    allow exceptions from the py-amqp library to be raised up to the
    application. This does not conform to the documented oslo.messaging
    API.
    
    Enhance the try except block to capture any exception and translate it
    into a MessageDeliveryFailure.
    
    There are a few cases where exceptions will be raised during autoretry
    publishing: recoverable connection or channel errors, and
    non-recoverable connection or channel errors.
    
    autoretry will only retry if the error is recoverable. Non recoverable
    errors are re-raised immediately regardless of the retry count.
    
    In the case of a recoverable error it seems unlikely that retrying
    either the connection or the channel yet again is going to get us
    anywhere, so in this case we simply clean up the channel state, log an
    error and fail the operation.
    
    In the case of non-recoverable error we are out of luck (think
    authentication failure) - further retrying will not help.  Best we can
    do is clean up state and log the heck out of it.
    
    Change-Id: I2f65d2ee19a8c3e9a323b30404abbf0cbb45a216
    Closes-Bug: #1705351
    Closes-Bug: #1707160


** Changed in: oslo.messaging
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1705351

Title:
  agent notifier getting amqp NotFound exceptions propagated up to it

Status in neutron:
  Fix Released
Status in oslo.messaging:
  Fix Released

Bug description:
  There appears to be a scenario where amqp NotFound exceptions are
  being propagated up through oslo messaging to the Neutron RPC client
  attempting to perform a cast.

  Since this appears in a grenade job, my guess is that it's due to the
  agents no longer listening on that particular exchange after they have
  been upgraded (the "port-delete" exchange is no longer used by agents
  in master).

  In neutron we probably need to ignore these exceptions on casts since
  it just means there are no longer agents interested in listening to
  the casts. However, this also seems like it may be an oslo messaging
  bug because in order for us to catch and ignore this in Neutron we
  would need to import AMQP directly for the exception. Directly
  depending on AMQP imports breaks the olso.messaging abstraction.

  From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-
  ironic-inspector-ubuntu-
  xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858
  :

  
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return super(Ml2Plugin, self).delete_network(context, id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.delete_port(context.elevated(), port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     context, port, router_ids, bound_mech_contexts)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.notifier.port_delete(context, port['id'])
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     cctxt.cast(context, 'port_delete', port_id=port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     timeout=timeout, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     conn.fanout_send(target.topic, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.ensure(method, retry=retry, error_callback=_error_callback)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ret, channel = autoretry_method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, channel=channels[0], **kwargs), channels[0]
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     compression=self.kombu_compression)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     exchange_name, declare,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     mandatory=mandatory, immediate=immediate,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.wait(spec.Basic.Ack)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.connection.drain_events(timeout=timeout)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     while not self.blocking_read(timeout):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.on_inbound_frame(frame)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     callback(channel, method_sig, buf, None)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method_sig, payload, content,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     listener(*args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     reply_code, reply_text, (class_id, method_id), ChannelError,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1705351/+subscriptions
References

[Bug 1705351] [NEW] agent notifier getting amqp NotFound exceptions propagated up to it
From: Kevin Benton, 2017-07-19