yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1701327] [NEW] While updating fixed_ip attribute of neutron port, get rabbit error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sridhar Venkat <svenkat@xxxxxxxxxx>
Date: Thu, 29 Jun 2017 17:16:08 -0000
Reply-to: Bug 1701327 <1701327@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

While attempting to update fixed_ip attribute of neutron port, get this error:
2017-06-29 02:55:58.856 37283 ERROR oslo.messaging._drivers.impl_rabbit [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] Failed to publish message to topic 'q-agent-notifier-l2population-update_fanout': Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'

Stack trace from neutron server log :
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] Mechanism driver 'l2population' failed in update_port_postcommit
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 408, in _call_on_drivers
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 156, in update_port_postcommit
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self._fixed_ips_changed(context, orig, port, diff_ips)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 134, in _fixed_ips_changed
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.rpc_ctx, {'chg_ip': upd_fdb_entries})
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py", line 87, in update_fdb_entries
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     fdb_entries)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py", line 49, in _notification_fanout
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     cctxt.cast(context, method, fdb_entries=fdb_entries)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 147, in cast
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     timeout=timeout, retry=retry)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     retry=retry)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 432, in _send
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     conn.fanout_send(target.topic, msg, retry=retry)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1295, in fanout_send
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1170, in _ensure_publishing
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.ensure(method, retry=retry, error_callback=_error_callback)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 817, in ensure
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     ret, channel = autoretry_method()
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return fun(*args, **kwargs)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return fun(*args, channel=channels[0], **kwargs), channels[0]
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 793, in execute_method
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     method()
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1201, in _publish
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     compression=self.kombu_compression)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     routing_key, mandatory, immediate, exchange, declare)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 188, in _publish
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     mandatory=mandatory, immediate=immediate,
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 2130, in basic_publish_confirm
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.wait([(60, 80)])
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return self.dispatch_method(method_sig, args, content)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return amqp_method(self, args)
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     reply_code, reply_text, (class_id, method_id), ChannelError,
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'
2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers
2017-06-29 02:55:58.903 37283 ERROR neutron.plugins.ml2.plugin [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] mechanism_manager.update_port_postcommit failed for port 799abd6f-c1d7-4fd3-8a77-65f1b0bf9515

In rabbitmq log
=ERROR REPORT==== 29-Jun-2017::04:21:48 ===
Channel error on connection <0.14694.1> (127.0.0.1:52206 -> 127.0.0.1:5671, vhost: '/', user: 'powervc_mq'), channel 1:
operation basic.publish caused a channel exception not_found: "no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'"
 
Version :

This is happening in Openstack ocata level environment. Two out of three
attempts result in this error. One of them went through succesfully
later on.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1701327

Title:
  While updating fixed_ip attribute of neutron port, get rabbit error

Status in neutron:
  New

Bug description:
  While attempting to update fixed_ip attribute of neutron port, get this error:
  2017-06-29 02:55:58.856 37283 ERROR oslo.messaging._drivers.impl_rabbit [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] Failed to publish message to topic 'q-agent-notifier-l2population-update_fanout': Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'

  Stack trace from neutron server log :
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] Mechanism driver 'l2population' failed in update_port_postcommit
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 408, in _call_on_drivers
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     getattr(driver.obj, method_name)(context)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 156, in update_port_postcommit
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self._fixed_ips_changed(context, orig, port, diff_ips)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 134, in _fixed_ips_changed
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.rpc_ctx, {'chg_ip': upd_fdb_entries})
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py", line 87, in update_fdb_entries
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     fdb_entries)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/l2pop/rpc.py", line 49, in _notification_fanout
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     cctxt.cast(context, method, fdb_entries=fdb_entries)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 147, in cast
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     timeout=timeout, retry=retry)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     retry=retry)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 432, in _send
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     conn.fanout_send(target.topic, msg, retry=retry)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1295, in fanout_send
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self._ensure_publishing(self._publish, exchange, msg, retry=retry)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1170, in _ensure_publishing
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.ensure(method, retry=retry, error_callback=_error_callback)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 817, in ensure
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     ret, channel = autoretry_method()
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return fun(*args, **kwargs)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return fun(*args, channel=channels[0], **kwargs), channels[0]
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 793, in execute_method
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     method()
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1201, in _publish
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     compression=self.kombu_compression)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     routing_key, mandatory, immediate, exchange, declare)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 188, in _publish
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     mandatory=mandatory, immediate=immediate,
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 2130, in basic_publish_confirm
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     self.wait([(60, 80)])
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return self.dispatch_method(method_sig, args, content)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     return amqp_method(self, args)
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers     reply_code, reply_text, (class_id, method_id), ChannelError,
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'
  2017-06-29 02:55:58.858 37283 ERROR neutron.plugins.ml2.managers
  2017-06-29 02:55:58.903 37283 ERROR neutron.plugins.ml2.plugin [req-2ea397ae-d5fd-4860-aca6-e633bba0d84a 0688b01e6439ca32d698d20789d52169126fb41fb1a4ddafcebb97d854e836c9 2d374653dc914c2e877fb6900ef28ccd - - -] mechanism_manager.update_port_postcommit failed for port 799abd6f-c1d7-4fd3-8a77-65f1b0bf9515

  In rabbitmq log
  =ERROR REPORT==== 29-Jun-2017::04:21:48 ===
  Channel error on connection <0.14694.1> (127.0.0.1:52206 -> 127.0.0.1:5671, vhost: '/', user: 'powervc_mq'), channel 1:
  operation basic.publish caused a channel exception not_found: "no exchange 'q-agent-notifier-l2population-update_fanout' in vhost '/'"
   
  Version :

  This is happening in Openstack ocata level environment. Two out of
  three attempts result in this error. One of them went through
  succesfully later on.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1701327/+subscriptions