yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1444201] Re: change in ipset elements breaks agent

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Tue, 21 Apr 2015 13:10:14 -0000
Reply-to: Bug 1444201 <1444201@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron/kilo
   Importance: Undecided
       Status: New

** Changed in: neutron/kilo
   Importance: Undecided => Critical

** Changed in: neutron/kilo
    Milestone: None => kilo-rc2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1444201

Title:
  change in ipset elements breaks agent

Status in OpenStack Neutron (virtual network service):
  Fix Committed
Status in neutron kilo series:
  New

Bug description:
  If an ipset element count changes (e.g. due to
  https://review.openstack.org/#/c/170328/), the previous elements will
  cause the create command to fail even though the -exist flag is being
  passed. This prevents the agent from setting up the ipsets correctly.

  Example exception:
  015-03-30 15:31:40.887 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Releasing file lock "/opt/stack/data/neutron/lock/neutron-
  ipset" after holding it for 0.005s release /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
  2015-03-30 15:31:40.888 DEBUG oslo_concurrency.lockutils [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Lock "ipset" released by "set_members" :: held 0.005s inne
  r /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
  2015-03-30 15:31:40.888 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-13ed77d8-6224-44ba-a2a9-becaa991663c None None] Error while processing VIF ports
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
  y", line 1586, in rpc_loop
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.p
  y", line 1350, in process_network_ports
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     port_info.get('updated', set()))
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 360, in se
  tup_port_filters
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.prepare_devices_filter(new_devices)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 219, in de
  corated_function
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     *args, **kwargs)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 244, in pr
  epare_devices_filter
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     security_groups, security_group_member_ips)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/firewall.py", line 106, in defer_apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 659,
  in filter_defer_apply_off
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.unfiltered_ports)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
    File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 155, in _setup_chains_apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._setup_chain(port, INGRESS_DIRECTION)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 182, in _setup_chain
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._add_rules_by_security_group(port, DIRECTION)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 411, in _add_rules_by_security_group
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._update_ipset_members(remote_sg_ids)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 448, in _update_ipset_members
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.ipset.set_members(sg_id, ip_version, current_ips)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 74, in set_members
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._create_set(set_name, ethertype)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 123, in _create_set
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/ipset_manager.py", line 132, in _apply
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.execute(cmd_ns, run_as_root=True, process_input=input)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['ipset', 'create', '-exist', u'IPv45d5e4955-d64c-40af-853e', 'hash:ip', 'family', 'inet', 'hashsize', '2048', 'maxelem', '131072']
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout:
  2015-03-30 15:31:40.888 32755 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.20.1: Set cannot be created: set with the same name already exists

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1444201/+subscriptions
References

[Bug 1444201] [NEW] change in ipset elements breaks agent
From: Kevin Benton, 2015-04-14