yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1705351] Re: agent notifier getting amqp NotFound exceptions propagated up to it

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1705351@xxxxxxxxxxxxxxxxxx>
Date: Fri, 04 Aug 2017 02:37:10 -0000
Reply-to: Bug 1705351 <1705351@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/490305
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0b5750ca518314a10266f25cb09997b7a859ab1e
Submitter: Jenkins
Branch:    master

commit 0b5750ca518314a10266f25cb09997b7a859ab1e
Author: Kevin Benton <blak111@xxxxxxxxx>
Date:   Wed Aug 2 20:14:10 2017 -0700

    Sinkhole workaround for old topics
    
    This patch introduces a workaround on the Neutron side
    for bug 1705351 that results in the Neutron server blocking
    trying to send to topics the agents no longer subscribe to.
    
    The workaround is to just subscribe to those topics and do
    nothing with the messages that come in until oslo.messaging
    can properly recover from loss of listeners.
    
    Change-Id: I946a33dfd0c0da26bb47b524f75f53bf59d3fbd5
    Closes-Bug: #1705351


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1705351

Title:
  agent notifier getting amqp NotFound exceptions propagated up to it

Status in neutron:
  Fix Released
Status in oslo.messaging:
  In Progress

Bug description:
  There appears to be a scenario where amqp NotFound exceptions are
  being propagated up through oslo messaging to the Neutron RPC client
  attempting to perform a cast.

  Since this appears in a grenade job, my guess is that it's due to the
  agents no longer listening on that particular exchange after they have
  been upgraded (the "port-delete" exchange is no longer used by agents
  in master).

  In neutron we probably need to ignore these exceptions on casts since
  it just means there are no longer agents interested in listening to
  the casts. However, this also seems like it may be an oslo messaging
  bug because in order for us to catch and ignore this in Neutron we
  would need to import AMQP directly for the exception. Directly
  depending on AMQP imports breaks the olso.messaging abstraction.

  From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-
  ironic-inspector-ubuntu-
  xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858
  :

  
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return super(Ml2Plugin, self).delete_network(context, id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.delete_port(context.elevated(), port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     context, port, router_ids, bound_mech_contexts)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.notifier.port_delete(context, port['id'])
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     cctxt.cast(context, 'port_delete', port_id=port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     timeout=timeout, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     conn.fanout_send(target.topic, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.ensure(method, retry=retry, error_callback=_error_callback)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ret, channel = autoretry_method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, channel=channels[0], **kwargs), channels[0]
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     compression=self.kombu_compression)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     exchange_name, declare,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     mandatory=mandatory, immediate=immediate,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.wait(spec.Basic.Ack)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.connection.drain_events(timeout=timeout)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     while not self.blocking_read(timeout):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.on_inbound_frame(frame)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     callback(channel, method_sig, buf, None)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method_sig, payload, content,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     listener(*args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     reply_code, reply_text, (class_id, method_id), ChannelError,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1705351/+subscriptions
References

[Bug 1705351] [NEW] agent notifier getting amqp NotFound exceptions propagated up to it
From: Kevin Benton, 2017-07-19