yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1491131] Re: Ipset race condition

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1491131@xxxxxxxxxxxxxxxxxx>
Date: Fri, 18 Sep 2015 09:25:37 -0000
Reply-to: Bug 1491131 <1491131@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: neutron
       Status: Invalid => In Progress

** Changed in: neutron
     Assignee: Cedric Brandily (cbrandily) => shihanzhang (shihanzhang)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1491131

Title:
  Ipset race condition

Status in neutron:
  In Progress
Status in neutron juno series:
  Confirmed
Status in neutron kilo series:
  Confirmed

Bug description:
  Hello,

  We have been using ipsets in neutron since juno.  We have upgraded our
  install to kilo a month or so and we have experienced 3 issues with
  ipsets.

  The issues are as follows:
  1.) Iptables attempts to apply rules for an ipset that was not added
  2.) iptables attempt to apply rules for an ipset that was removed, but still refrenced in the iptables config
  3.) ipset churns trying to remove an ipset that has already been removed.

  For issue one and two I am unable to get the logs for these issues
  because neutron was dumping the full iptables-restore entries to log
  once every second for a few hours and eventually filled up the disk
  and we removed the file to get things working again.

  For issue 3.) I have the start of the logs here:
  2015-08-31 12:17:00.100 6840 INFO neutron.agent.common.ovs_lib [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Port 29355e52-bae1-44b2-ace6-5bc7ce497d32 not present in bridge br-int
  2015-08-31 12:17:00.101 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:00.101 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Remove device filter for [u'2aa0f79d-4983-4c7a-b489-e0612c482e36']
  2015-08-31 12:17:00.861 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port 2aa0f79d-4983-4c7a-b489-e0612c482e36 not present in bridge br-int
  2015-08-31 12:17:00.862 4581 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:00.862 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'c616328a-e44c-4cf8-bc8e-83058c5635dd']
  2015-08-31 12:17:01.499 4581 INFO neutron.agent.securitygroups_rpc [req-b5b95389-52b6-4051-ab35-ae383df56a0b ] Security group member updated [u'b05f4fa6-f1ec-41c0-8ba6-80b859dc23b0']
  2015-08-31 12:17:01.500 6840 INFO neutron.agent.securitygroups_rpc [req-b5b95389-52b6-4051-ab35-ae383df56a0b ] Security group member updated [u'b05f4fa6-f1ec-41c0-8ba6-80b859dc23b0']
  2015-08-31 12:17:01.608 6840 INFO neutron.agent.common.ovs_lib [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Port 2aa0f79d-4983-4c7a-b489-e0612c482e36 not present in bridge br-int
  2015-08-31 12:17:01.609 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:01.609 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Remove device filter for [u'c616328a-e44c-4cf8-bc8e-83058c5635dd']
  2015-08-31 12:17:02.358 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port c616328a-e44c-4cf8-bc8e-83058c5635dd not present in bridge br-int
  2015-08-31 12:17:02.359 4581 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:02.359 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'fddff586-9903-47ad-92e1-b334e02e9d1c']
  2015-08-31 12:17:03.108 6840 INFO neutron.agent.common.ovs_lib [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Port c616328a-e44c-4cf8-bc8e-83058c5635dd not present in bridge br-int
  2015-08-31 12:17:03.109 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:03.109 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Remove device filter for [u'fddff586-9903-47ad-92e1-b334e02e9d1c']
  2015-08-31 12:17:03.855 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port fddff586-9903-47ad-92e1-b334e02e9d1c not present in bridge br-int
  2015-08-31 12:17:03.855 4581 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:03.856 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'3f706749-f8bb-41ab-aa4c-a0925dc67bd4']
  2015-08-31 12:17:03.919 4581 INFO neutron.agent.securitygroups_rpc [req-1872b212-b537-41cc-96af-0c6ad380824c ] Security group member updated [u'b05f4fa6-f1ec-41c0-8ba6-80b859dc23b0']
  2015-08-31 12:17:04.166 6840 INFO neutron.agent.securitygroups_rpc [req-1872b212-b537-41cc-96af-0c6ad380824c ] Security group member updated [u'b05f4fa6-f1ec-41c0-8ba6-80b859dc23b0']
  2015-08-31 12:17:04.620 6840 INFO neutron.agent.common.ovs_lib [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Port fddff586-9903-47ad-92e1-b334e02e9d1c not present in bridge br-int
  2015-08-31 12:17:04.621 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:04.621 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Remove device filter for [u'3f706749-f8bb-41ab-aa4c-a0925dc67bd4']
  2015-08-31 12:17:05.422 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port 3f706749-f8bb-41ab-aa4c-a0925dc67bd4 not present in bridge br-int
  2015-08-31 12:17:05.422 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'f075fda0-2dfc-4c1b-b7af-66a556b32434']
  2015-08-31 12:17:05.674 4581 INFO neutron.agent.securitygroups_rpc [req-2aa43ed7-6749-4313-a6bd-d46ec8fc4d29 ] Security group member updated [u'7ad253a3-71ae-4539-a973-b9afb0f23e3b']
  2015-08-31 12:17:05.679 6840 INFO neutron.agent.securitygroups_rpc [req-2aa43ed7-6749-4313-a6bd-d46ec8fc4d29 ] Security group member updated [u'7ad253a3-71ae-4539-a973-b9afb0f23e3b']
  2015-08-31 12:17:06.037 6840 ERROR neutron.agent.linux.utils [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] 
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'destroy', 'NETIPv420e1594a-28bb-4591-b']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: ipset v6.11: The set with the given name does not exist

  2015-08-31 12:17:06.040 6840 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Error while processing VIF ports
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1589, in rpc_loop
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.process_deleted_ports(port_info)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 355, in process_deleted_ports
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.remove_devices_filter([port_id])
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 303, in remove_devices_filter
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.remove_port_filter(device)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/firewall.py", line 106, in defer_apply
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 660, in filter_defer_apply_off
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._remove_unused_security_group_info()
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 583, in _remove_unused_security_group_info
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._remove_ipsets_for_remote_sgs(ip_version, remote_sg_ids)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 642, in _remove_ipsets_for_remote_sgs
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.ipset.destroy(remote_sg_id, ip_version)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 101, in destroy
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._destroy(set_name, forced)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 174, in _destroy
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 138, in _apply
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.execute(cmd_ns, run_as_root=True, process_input=input)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'destroy', 'NETIPv420e1594a-28bb-4591-b']
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin: 
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: 
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.11: The set with the given name does not exist
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
  2015-08-31 12:17:06.040 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
  2015-08-31 12:17:06.042 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Agent out of sync with plugin!
  2015-08-31 12:17:06.045 4581 INFO neutron.agent.securitygroups_rpc [req-75496835-97a3-4d3c-a2ad-7fe3ce7d88a2 ] Security group member updated [u'ba07732e-88fb-4bd1-a018-99221d85c283', u'e716ee7f-df77-4722-b117-dbb6907b2319']
  2015-08-31 12:17:06.053 6840 INFO neutron.agent.securitygroups_rpc [req-75496835-97a3-4d3c-a2ad-7fe3ce7d88a2 ] Security group member updated [u'ba07732e-88fb-4bd1-a018-99221d85c283', u'e716ee7f-df77-4722-b117-dbb6907b2319']
  2015-08-31 12:17:06.488 6840 INFO neutron.agent.securitygroups_rpc [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Remove device filter for [u'0457636d-0ae6-412a-8025-6579e1f8a78a']
  2015-08-31 12:17:06.748 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port f075fda0-2dfc-4c1b-b7af-66a556b32434 not present in bridge br-int
  2015-08-31 12:17:06.749 4581 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:06.749 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'cbd00d69-4069-4d65-bc5d-25cfc57ae9b5']
  2015-08-31 12:17:07.474 6840 ERROR neutron.agent.linux.utils [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] 
  Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'destroy', 'NETIPv420e1594a-28bb-4591-b']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: ipset v6.11: The set with the given name does not exist

  2015-08-31 12:17:07.475 6840 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Error while processing VIF ports
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1589, in rpc_loop
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.process_deleted_ports(port_info)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 355, in process_deleted_ports
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.remove_devices_filter([port_id])
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 303, in remove_devices_filter
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.remove_port_filter(device)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/rh/python27/root/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/firewall.py", line 106, in defer_apply
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 660, in filter_defer_apply_off
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._remove_unused_security_group_info()
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 583, in _remove_unused_security_group_info
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._remove_ipsets_for_remote_sgs(ip_version, remote_sg_ids)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/iptables_firewall.py", line 642, in _remove_ipsets_for_remote_sgs
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.ipset.destroy(remote_sg_id, ip_version)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 101, in destroy
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._destroy(set_name, forced)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 174, in _destroy
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply(cmd)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/ipset_manager.py", line 138, in _apply
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.execute(cmd_ns, run_as_root=True, process_input=input)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/openstack/neutron/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ipset', 'destroy', 'NETIPv420e1594a-28bb-4591-b']
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 1
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdin: 
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: 
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: ipset v6.11: The set with the given name does not exist
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
  2015-08-31 12:17:07.475 6840 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
  2015-08-31 12:17:08.042 6840 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-2505ffdc-85a7-46f9-bb0f-3fb7fe3d3eed ] Agent out of sync with plugin!
  2015-08-31 12:17:08.177 4581 INFO neutron.agent.common.ovs_lib [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Port cbd00d69-4069-4d65-bc5d-25cfc57ae9b5 not present in bridge br-int
  2015-08-31 12:17:08.177 4581 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] port_unbound(): net_uuid None not in local_vlan_map
  2015-08-31 12:17:08.178 4581 INFO neutron.agent.securitygroups_rpc [req-4144d47e-0044-47b3-a302-d019e0a67aa0 ] Remove device filter for [u'4ddf7f2a-deec-448e-9378-3fbc95172774']

  
  This bascially churns like this until neutron-openvsiwtch-agent is restarted and then everything is fine.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1491131/+subscriptions
References

[Bug 1491131] [NEW] Ipset race condition - Kilo-2015.1.0
From: Kris Lindgren, 2015-09-01