yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1705351] [NEW] agent notifier getting amqp NotFound exceptions propagated up to it

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1705351@xxxxxxxxxxxxxxxxxx>
Date: Wed, 19 Jul 2017 21:49:39 -0000
Reply-to: Bug 1705351 <1705351@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

There appears to be a scenario where amqp NotFound exceptions are being
propagated up through oslo messaging to the Neutron RPC client
attempting to perform a cast.

Since this appears in a grenade job, my guess is that it's due to the
agents no longer listening on that particular exchange after they have
been upgraded (the "port-delete" exchange is no longer used by agents in
master).

In neutron we probably need to ignore these exceptions on casts since it
just means there are no longer agents interested in listening to the
casts. However, this also seems like it may be an oslo messaging bug
because in order for us to catch and ignore this in Neutron we would
need to import AMQP directly for the exception. Directly depending on
AMQP imports breaks the olso.messaging abstraction.

>From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-
ironic-inspector-ubuntu-
xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858
:


2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return super(Ml2Plugin, self).delete_network(context, id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.delete_port(context.elevated(), port_id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     context, port, router_ids, bound_mech_contexts)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.notifier.port_delete(context, port['id'])
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     cctxt.cast(context, 'port_delete', port_id=port_id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     timeout=timeout, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     conn.fanout_send(target.topic, msg, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.ensure(method, retry=retry, error_callback=_error_callback)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ret, channel = autoretry_method()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, channel=channels[0], **kwargs), channels[0]
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     compression=self.kombu_compression)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     exchange_name, declare,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     mandatory=mandatory, immediate=immediate,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.wait(spec.Basic.Ack)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.connection.drain_events(timeout=timeout)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     while not self.blocking_read(timeout):
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.on_inbound_frame(frame)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     callback(channel, method_sig, buf, None)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method_sig, payload, content,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     listener(*args)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     reply_code, reply_text, (class_id, method_id), ChannelError,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation

** Affects: neutron
     Importance: High
         Status: Confirmed

** Affects: oslo.messaging
     Importance: Undecided
         Status: New

** Changed in: neutron
       Status: New => Confirmed

** Changed in: neutron
   Importance: Undecided => High

** Also affects: oslo.messaging
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1705351

Title:
  agent notifier getting amqp NotFound exceptions propagated up to it

Status in neutron:
  Confirmed
Status in oslo.messaging:
  New

Bug description:
  There appears to be a scenario where amqp NotFound exceptions are
  being propagated up through oslo messaging to the Neutron RPC client
  attempting to perform a cast.

  Since this appears in a grenade job, my guess is that it's due to the
  agents no longer listening on that particular exchange after they have
  been upgraded (the "port-delete" exchange is no longer used by agents
  in master).

  In neutron we probably need to ignore these exceptions on casts since
  it just means there are no longer agents interested in listening to
  the casts. However, this also seems like it may be an oslo messaging
  bug because in order for us to catch and ignore this in Neutron we
  would need to import AMQP directly for the exception. Directly
  depending on AMQP imports breaks the olso.messaging abstraction.

  From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-
  ironic-inspector-ubuntu-
  xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858
  :

  
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.plugin_deleter(*deleter_args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return super(Ml2Plugin, self).delete_network(context, id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.delete_port(context.elevated(), port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(self, context, *args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return method(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     context, port, router_ids, bound_mech_contexts)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.notifier.port_delete(context, port['id'])
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     cctxt.cast(context, 'port_delete', port_id=port_id)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     timeout=timeout, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     conn.fanout_send(target.topic, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.ensure(method, retry=retry, error_callback=_error_callback)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     ret, channel = autoretry_method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, **kwargs)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return fun(*args, channel=channels[0], **kwargs), channels[0]
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method()
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     compression=self.kombu_compression)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     exchange_name, declare,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     mandatory=mandatory, immediate=immediate,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.wait(spec.Basic.Ack)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     self.connection.drain_events(timeout=timeout)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     while not self.blocking_read(timeout):
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     return self.on_inbound_frame(frame)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     callback(channel, method_sig, buf, None)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     method_sig, payload, content,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     listener(*args)
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation     reply_code, reply_text, (class_id, method_id), ChannelError,
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
  2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1705351/+subscriptions
Follow ups

[Bug 1705351] Re: agent notifier getting amqp NotFound exceptions propagated up to it
From: OpenStack Infra, 2017-08-22
[Bug 1705351] Re: agent notifier getting amqp NotFound exceptions propagated up to it
From: OpenStack Infra, 2017-08-04