yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1854050] Re: minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nate Johnston <1854050@xxxxxxxxxxxxxxxxxx>
Date: Tue, 26 Nov 2019 14:23:47 -0000
Reply-to: Bug 1854050 <1854050@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1854050

Title:
  minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha

Status in neutron:
  Invalid

Bug description:
  Environment is neutron 14.0.2 with DVR and HA (OVS).
  Upgraded a single compute or deployed new with 14.0.3.

  Expected outcome:

  Minor versions should be fully compatible and neutron should work with
  the same major version.

  Actual outcome:

  Can't schedule instances on computes holding this version and neutron
  services spew out errors.

  neutron-server on controller/network node:

  Exception during message handling: InvalidTargetVersion: Invalid target version 1.5
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     **filter_kwargs)]
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server     raise exception.InvalidTargetVersion(version=target_version)
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server InvalidTargetVersion: Invalid target version 1.5
  2019-11-26 08:36:51.359 25 ERROR oslo_messaging.rpc.server 

  
  neutron-openvswitch-agent on compute node:

  Error while processing VIF ports: RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
  [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n    return func(*args, **kwargs)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n    **filter_kwargs)]\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2278, in rpc_loop
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1845, in process_network_ports
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._apply_port_filter(device_ids)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 143, in _apply_port_filter
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.context, list(device_ids)[i:i + step])
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py", line 232, in security_group_info_for_devices
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ports = self._get_devices_info(context, devices)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py", line 300, in _get_devices_info
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ovo = self.rcache.get_resource_by_id('Port', device)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 61, in get_resource_by_id
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_restarted=agent_restarted)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 79, in _flood_cache_for_query
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     filter_kwargs=filter_kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return method(*args, **kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 114, in bulk_pull
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py", line 157, in call
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     call_monitor_timeout, retry=retry)
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise result
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n    return func(*args, **kwargs)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n    **filter_kwargs)]\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
  2019-11-26 08:36:45.975 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 

  AND

  Failed to get details for device 227d33c9-a8c5-45aa-bd98-8bc3ff2f6029: RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
  [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n    return func(*args, **kwargs)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n    **filter_kwargs)]\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc Traceback (most recent call last):
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 303, in get_devices_details_list_and_failed_devices
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     agent_restarted))
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/rpc.py", line 312, in get_device_details
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     resources.PORT, device, agent_restarted)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 61, in get_resource_by_id
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     agent_restarted=agent_restarted)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 79, in _flood_cache_for_query
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     filter_kwargs=filter_kwargs)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     return method(*args, **kwargs)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 114, in bulk_pull
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron_lib/rpc.py", line 157, in call
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     return self._original_context.call(ctxt, method, **kwargs)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     retry=self.retry)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 128, in _send
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     retry=retry)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     call_monitor_timeout, retry=retry)
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc     raise result
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc RemoteError: Remote error: InvalidTargetVersion Invalid target version 1.5
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc [u'Traceback (most recent call last):\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner\n    return func(*args, **kwargs)\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 148, in bulk_pull\n    **filter_kwargs)]\n', u'  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 551, in obj_to_primitive\n    raise exception.InvalidTargetVersion(version=target_version)\n', u'InvalidTargetVersion: Invalid target version 1.5\n'].
  2019-11-26 08:36:45.907 6 ERROR neutron.agent.rpc 

  
  AND

  Clearing cache of registered ports, retries to resync were > 5:
  RemoteError: Remote error: InvalidTargetVersion Invalid target version
  1.5


  I have switched back to 14.0.2 (5ish weeks old) and things are back on
  track in the environment.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1854050/+subscriptions
References

[Bug 1854050] [NEW] minor versions 14.0.2 & 14.0.3 are not compatible in dvr-ha
From: Marek Grudzinski, 2019-11-26