yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1567507] [NEW] neutron-lbaas broken with neutron change

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rabi Mishra <1567507@xxxxxxxxxxxxxxxxxx>
Date: Thu, 07 Apr 2016 15:28:02 -0000
Reply-to: Bug 1567507 <1567507@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

It seems recent change
https://github.com/openstack/neutron/commit/34a328fe12950c339b8259451262470c627f2f00
has broken neutron-lbaas.

Hence all dependent projects are broken with below error in q-lbaas.

2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [req-0a3a7771-0f1e-4424-9b96-0b7613cc1c82 demo -] Create vip 7c347fc8-c282-4231-aa1c-e23a0d180abb failed on device driver haproxy_ns
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/agent/agent_manager.py", line 227, in create_vip
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     driver.create_vip(vip)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 348, in create_vip
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self._refresh_device(vip['pool_id'])
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 344, in _refresh_device
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     if not self.deploy_instance(logical_config) and self.exists(pool_id):
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     return f(*args, **kwargs)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 337, in deploy_instance
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self.create(logical_config)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 92, in create
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     logical_config['vip']['address'])
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 247, in _plug
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self.plugin_rpc.plug_vip_port(port['id'])
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/agent/agent_api.py", line 58, in plug_vip_port
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     host=self.host)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=self.retry)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=retry)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     raise result
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager AttributeError: 'str' object has no attribute 'strftime'
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     incoming.message))
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     return self._do_dispatch(endpoint, method, ctxt, args)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result = func(ctxt, **new_args)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/common/agent_driver_base.py", line 200, in plug_vip_port
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     {'port': port}
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1210, in update_port
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     port)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1301, in update_port
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result = self._make_port_dict(port)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_common.py", line 189, in _make_port_dict
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     attributes.PORTS, res, port)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/common_db_mixin.py", line 271, in _apply_dict_extend_functions
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     func(*args)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/services/timestamp/timestamp_db.py", line 103, in extend_resource_dict_timestamp
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self._format_timestamp(resource_db, resource_res)
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/services/timestamp/timestamp_db.py", line 94, in _format_timestamp
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result['created_at'] = (resource_db.created_at.
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager AttributeError: 'str' object has no attribute 'strftime'
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager

** Affects: neutron
     Importance: Critical
         Status: New


** Tags: gate-failure lbaas

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1567507

Title:
  neutron-lbaas broken with neutron change

Status in neutron:
  New

Bug description:
  It seems recent change
  https://github.com/openstack/neutron/commit/34a328fe12950c339b8259451262470c627f2f00
  has broken neutron-lbaas.

  Hence all dependent projects are broken with below error in q-lbaas.

  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager [req-0a3a7771-0f1e-4424-9b96-0b7613cc1c82 demo -] Create vip 7c347fc8-c282-4231-aa1c-e23a0d180abb failed on device driver haproxy_ns
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/agent/agent_manager.py", line 227, in create_vip
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     driver.create_vip(vip)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 348, in create_vip
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self._refresh_device(vip['pool_id'])
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 344, in _refresh_device
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     if not self.deploy_instance(logical_config) and self.exists(pool_id):
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     return f(*args, **kwargs)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 337, in deploy_instance
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self.create(logical_config)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 92, in create
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     logical_config['vip']['address'])
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 247, in _plug
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self.plugin_rpc.plug_vip_port(port['id'])
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/agent/agent_api.py", line 58, in plug_vip_port
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     host=self.host)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=self.retry)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     timeout=timeout, retry=retry)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     retry=retry)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     raise result
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager AttributeError: 'str' object has no attribute 'strftime'
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager Traceback (most recent call last):
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     incoming.message))
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result = func(ctxt, **new_args)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/drivers/common/agent_driver_base.py", line 200, in plug_vip_port
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     {'port': port}
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1210, in update_port
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     port)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1301, in update_port
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result = self._make_port_dict(port)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/db_base_plugin_common.py", line 189, in _make_port_dict
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     attributes.PORTS, res, port)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/db/common_db_mixin.py", line 271, in _apply_dict_extend_functions
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     func(*args)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/services/timestamp/timestamp_db.py", line 103, in extend_resource_dict_timestamp
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     self._format_timestamp(resource_db, resource_res)
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager   File "/opt/stack/new/neutron/neutron/services/timestamp/timestamp_db.py", line 94, in _format_timestamp
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager     result['created_at'] = (resource_db.created_at.
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager AttributeError: 'str' object has no attribute 'strftime'
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager 
  2016-04-07 13:47:56.319 28677 ERROR neutron_lbaas.services.loadbalancer.agent.agent_manager

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1567507/+subscriptions
Follow ups

[Bug 1567507] Re: neutron-lbaas broken with neutron change
From: OpenStack Infra, 2016-04-08