← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #31929

[Bug 1444201] [NEW] change in ipset elements breaks agent

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

If an ipset element count changes (e.g. due to
https://review.openstack.org/#/c/170328/), the previous elements will
cause the create command to fail even though the -exist flag is being
passed. This prevents the agent from setting up the ipsets correctly.

Example exception:
015-03-30 15:31:40.887 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Releasing file lock "/opt/stack/data/neutron/lock/neutron-
ipset" after holding it for 0.005s release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-03-30 15:31:40.888 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Lock "ipset" released by "set_members" :: held 0.005s inne
r /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
2015-03-30 15:31:40.888 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Error while processing VIF ports
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
y", line 1586, in rpc_loop
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
y", line 1350, in process_network_ports
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 360, in se
tup_port_filters
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 219, in de
corated_function
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 244, in pr
epare_devices_filter
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     security_groups, security_group_member_ips)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/firewall.py", line 106, in defer_apply
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 659,
in filter_defer_apply_off
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.unfiltered_ports)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
  File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 155, in _setup_chains_apply
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._setup_chain(port, INGRESS_DIRECTION)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 182, in _setup_chain
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._add_rules_by_security_group(port, DIRECTION)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 411, in _add_rules_by_security_group
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._update_ipset_members(remote_sg_ids)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 448, in _update_ipset_members
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.ipset.set_members(sg_id, ip_version, current_ips)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 74, in set_members
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._create_set(set_name, ethertype)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 123, in _create_set
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 132, in _apply
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.execute(cmd_ns, run_as_root=True, process_input=input)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['ipset', 'create', '-exist', u'IPv45d5e4955-d64c-40af-853e', 'hash:ip', 'family', 'inet', 'hashsize', '2048', 'maxelem', '131072']
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin:
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout:
2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.20.1: Set cannot be created: set with the same name already exists

** Affects: neutron
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: New


** Tags: kilo-rc-potential

** Changed in: neutron
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

** Tags added: kilo-rc-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1444201

Title:
  change in ipset elements breaks agent

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  If an ipset element count changes (e.g. due to
  https://review.openstack.org/#/c/170328/), the previous elements will
  cause the create command to fail even though the -exist flag is being
  passed. This prevents the agent from setting up the ipsets correctly.

  Example exception:
  015-03-30 15:31:40.887 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Releasing file lock "/opt/stack/data/neutron/lock/neutron-
  ipset" after holding it for 0.005s release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
  2015-03-30 15:31:40.888 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Lock "ipset" released by "set_members" :: held 0.005s inne
  r /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
  2015-03-30 15:31:40.888 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Error while processing VIF ports
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
  y", line 1586, in rpc_loop
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
  y", line 1350, in process_network_ports
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 360, in se
  tup_port_filters
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 219, in de
  corated_function
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 244, in pr
  epare_devices_filter
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     security_groups, security_group_member_ips)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/firewall.py", line 106, in defer_apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 659,
  in filter_defer_apply_off
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.unfiltered_ports)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
    File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 155, in _setup_chains_apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._setup_chain(port, INGRESS_DIRECTION)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 182, in _setup_chain
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._add_rules_by_security_group(port, DIRECTION)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 411, in _add_rules_by_security_group
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._update_ipset_members(remote_sg_ids)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 448, in _update_ipset_members
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.ipset.set_members(sg_id, ip_version, current_ips)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 74, in set_members
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._create_set(set_name, ethertype)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 123, in _create_set
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 132, in _apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.execute(cmd_ns, run_as_root=True, process_input=input)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['ipset', 'create', '-exist', u'IPv45d5e4955-d64c-40af-853e', 'hash:ip', 'family', 'inet', 'hashsize', '2048', 'maxelem', '131072']
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.20.1: Set cannot be created: set with the same name already exists

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1444201/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next