yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1801104] [NEW] Migration causes timeout while doing bulk_pull

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Paul Peereboom <1801104@xxxxxxxxxxxxxxxxxx>
Date: Thu, 01 Nov 2018 14:55:31 -0000
Reply-to: Bug 1801104 <1801104@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I'm trying to figure out why some live-migrations in our environment
causes unreachability of the instance for about ~3 minutes.

When setting neutron-openvswitch in debug mode we can see that the
bulk_pull of resources takes about 2-3 minutes before retrieving about
7057 ports(!). While doing this query the instance is not reachable.

After the instance is migrated to the compute and is pingable again
after 3 minutes we can successfully migrate the instance back and forth
without downtime on the compute node. It looks like the cache is already
filled the second migration and a bulk_pull is not required again. After
some time it looks like the cache needs to be refreshed and a live-
migration causes the instance to be unreachable for minutes again. We
can clear the cache aswell when doing a restart of neutron-openvswitch-
agent.

I was able to reproduce this by adding a security group rule with any
protocol with a remote-group defined and attach that sec group to an
instance. Than migrate the instance to a compute node where neutron-
openvswitch is restarted and no other instance is running on it.


Info and logs:
Openstack version: Pike
OS: Redhat

instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6
port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b

Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration
Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected


[root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                                           |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at      | 2017-10-11T13:13:48Z                                                                                                                                                                                                                                                            |
| description     | Default security group                                                                                                                                                                                                                                                          |
| id              | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3                                                                                                                                                                                                                                            |
| name            | default                                                                                                                                                                                                                                                                         |
| project_id      | b9bd8eac1587429aa07ae00661f2a07b                                                                                                                                                                                                                                                |
| revision_number | 16                                                                                                                                                                                                                                                                              |                                                                                                 |
| updated_at      | 2018-04-17T08:14:55Z                                                                                                                                                                                                                                                            |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


[root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                                                                                                                                                                                      |
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at      | 2018-11-01T12:25:10Z                                                                                                                                                                                                                                       |
| description     |                                                                                                                                                                                                                                                            |
| id              | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1                                                                                                                                                                                                                       |
| location        | None                                                                                                                                                                                                                                                       |
| name            | temp_sec                                                                                                                                                                                                                                                   |
| project_id      | b9bd8eac1587429aa07ae00661f2a07b                                                                                                                                                                                                                           |
| revision_number | 10                                                                                                                                                                                                                                                         |                                                                                                   |
| tags            | []                                                                                                                                                                                                                                                         |
| updated_at      | 2018-11-01T13:22:21Z                                                                                                                                                                                                                                       |
+-----------------+-----------------------------------------------------+

#Default security group, Broken(downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 366a55ae-5086-4baf-a55c-bfef0cc38081 | None        | None              |            | None                                 |
| 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
| 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp        | 0.0.0.0/0         |            | None                                 |
| b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None        | None              |            | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None        | None              |            | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
| fdfad4d8-95bf-419a-aa17-40ad048da7bf | None        | None              |            | None                                 |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+

#temp_sec security group, Works(No downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp        | 0.0.0.0/0         |            | None                                 |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None        | None              |            | None                                 |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None        | None              |            | None                                 |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+


When I migrate an instance with default sec group attached to a compute
it's unavailable untill all the ports are retrieved with bulk query
(timestamp reachable again: 13:00:03.783):

#########I've added a log rule for the filter and resource type and query_id#########
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70
2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48
70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

<snip>

2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options=
[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id
=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set(
[2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413'])

2018-11-01 12:58:51.447 291758 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f
19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i
d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e
:08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po
licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15
1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i
d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e
:0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager [req-
08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f
19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks
['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854210276',
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387']
for Port, after_update _notify_loop /usr/lib/python2.7/site-
packages/neutron_lib/callbacks/manager.py:167

2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

2018-11-01 12:58:54.804 291758 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f
19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG: resource ID:
004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

######################## 
Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs
########################

2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache [req-
6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources returned
for queries set([('Port', ('security_group_ids',
('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92


#########################
When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating:
#########################

2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter:
{'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:69

2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70

2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: query_id:
set([('SecurityGroupRule', ('security_group_id', (u'13f878ef-
eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

2018-11-01 12:54:44.130 290907 DEBUG
neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c-
94c4a7f7476e - - - - -]
neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
bulk_pull called with arguments (<neutron_lib.context.Context object at
0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs':
{'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}}
wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66

2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
1831db02-7538-42ec-b04b-4c8c3d06dadb _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
1c49d1ad-a6f1-470b-bd99-36f32932a6e1 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
3c1d82af-acd7-4543-8922-a96c0c67db67 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
452928bf-d5eb-41ab-b520-f34c33c63058 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
4e9201db-3583-42c7-9c79-372a3394a159 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
4ecc1fa6-d090-4d11-9a3d-323e1f012d40 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
65264a84-ae99-4c90-8626-6dbac869d15d _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
677d7b02-8943-4824-8fa5-990a02503f03 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
d435b0d1-67bb-4834-9518-5bdf8f843610 _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache [req-
c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resource ID:
fc12d72c-94c9-432b-8f14-79412419c7db _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88

2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u
'13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

#########It is returning just 11 resources which is correct
value.###############


######### 
I can break the working sec group by adding a security group rule with any protocol with a remote-group defined:
#########

[root@os-admin (openstack_admin)]# openstack security group rule create
--remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol any
--project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b-
46f8-ac30-f13e48f9d4b1

#Remote group is added:
[root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp        | 0.0.0.0/0         |            | None                                 |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None        | None              |            | None                                 |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
| b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None        | None              |            | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None        | None              |            | None                                 |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+


2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69

2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-
24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resourcetype:
Port _flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:70

2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-
24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: query_id:
set([('Port', ('security_group_ids', ('5958c3ad-5d2b-
46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7
/site-packages/neutron/agent/resource_cache.py:72

2018-11-01 14:30:24.242 311475 DEBUG
neutron.api.rpc.handlers.resources_rpc [req-
24595a6d-9784-4676-b825-e4e41150c0ec - - - - -]
neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
bulk_pull called with arguments (<neutron_lib.context.Context object at
0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids':
('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7
/site-packages/oslo_log/helpers.py:66

2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache [req-
643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update
without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b
record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache [req-
dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port eb9288fe-
9c78-447c-ba77-e2cfd8df624b updated (revision_number 826->828). Old
fields: {'status': u'DOWN', 'binding':
PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding':
PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager [req-
dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:24.956 311475 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message
processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache [req-
643cbd7e-06e9-4576-9e1e-086b86c1726d a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update
without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b
record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache [req-
347c971e-3e22-4805-9b95-9a57d75f98aa a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Received resource Port update
without any changes: eb9288fe-9c78-447c-ba77-e2cfd8df624b
record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:189

2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache [req-
d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port eb9288fe-
9c78-447c-ba77-e2cfd8df624b updated (revision_number 829->830). Old
fields: {'status': u'ACTIVE'} New fields: {'status': u'DOWN'}
record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager [req-
d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:28.381 311475 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message
processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache [req-
94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78
-447c-ba77-e2cfd8df624b updated (revision_number 830->831). Old fields:
{'binding': PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-
9c78-447c-ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} New fields: {'binding':
PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache [req-
94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Resource Port eb9288fe-9c78
-447c-ba77-e2cfd8df624b updated (revision_number 831->832). Old fields:
{'binding': PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-
9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} New fields: {'binding':
PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197

2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager [req-
94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643
4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G
<snip>

#######
7060 resources matched!:
#######
2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b']
_apply_port_filter /usr/lib/python2.7/site-
packages/neutron/agent/securitygroups_rpc.py:135

2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
group information _update_security_group_info /usr/lib/python2.7/site-
packages/neutron/agent/securitygroups_rpc.py:149


I'm not sure why this behave this way, can someone help me with this issue?

Regards,
Paul Peereboom
ODC-Noord

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1801104

Title:
  Migration causes timeout while doing bulk_pull

Status in neutron:
  New

Bug description:
  I'm trying to figure out why some live-migrations in our environment
  causes unreachability of the instance for about ~3 minutes.

  When setting neutron-openvswitch in debug mode we can see that the
  bulk_pull of resources takes about 2-3 minutes before retrieving about
  7057 ports(!). While doing this query the instance is not reachable.

  After the instance is migrated to the compute and is pingable again
  after 3 minutes we can successfully migrate the instance back and
  forth without downtime on the compute node. It looks like the cache is
  already filled the second migration and a bulk_pull is not required
  again. After some time it looks like the cache needs to be refreshed
  and a live-migration causes the instance to be unreachable for minutes
  again. We can clear the cache aswell when doing a restart of neutron-
  openvswitch-agent.

  I was able to reproduce this by adding a security group rule with any
  protocol with a remote-group defined and attach that sec group to an
  instance. Than migrate the instance to a compute node where neutron-
  openvswitch is restarted and no other instance is running on it.

  
  Info and logs:
  Openstack version: Pike
  OS: Redhat

  instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6
  port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b

  Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration
  Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected

  
  [root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 
  +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field           | Value                                                                                                                                                                                                                                                                           |
  +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | created_at      | 2017-10-11T13:13:48Z                                                                                                                                                                                                                                                            |
  | description     | Default security group                                                                                                                                                                                                                                                          |
  | id              | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3                                                                                                                                                                                                                                            |
  | name            | default                                                                                                                                                                                                                                                                         |
  | project_id      | b9bd8eac1587429aa07ae00661f2a07b                                                                                                                                                                                                                                                |
  | revision_number | 16                                                                                                                                                                                                                                                                              |                                                                                                 |
  | updated_at      | 2018-04-17T08:14:55Z                                                                                                                                                                                                                                                            |
  +-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  
  [root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
  +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field           | Value                                                                                                                                                                                                                                                      |
  +-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | created_at      | 2018-11-01T12:25:10Z                                                                                                                                                                                                                                       |
  | description     |                                                                                                                                                                                                                                                            |
  | id              | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1                                                                                                                                                                                                                       |
  | location        | None                                                                                                                                                                                                                                                       |
  | name            | temp_sec                                                                                                                                                                                                                                                   |
  | project_id      | b9bd8eac1587429aa07ae00661f2a07b                                                                                                                                                                                                                           |
  | revision_number | 10                                                                                                                                                                                                                                                         |                                                                                                   |
  | tags            | []                                                                                                                                                                                                                                                         |
  | updated_at      | 2018-11-01T13:22:21Z                                                                                                                                                                                                                                       |
  +-----------------+-----------------------------------------------------+

  #Default security group, Broken(downtime)
  [root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | 366a55ae-5086-4baf-a55c-bfef0cc38081 | None        | None              |            | None                                 |
  | 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
  | 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp        | 0.0.0.0/0         |            | None                                 |
  | b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None        | None              |            | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
  | bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None        | None              |            | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
  | e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
  | fdfad4d8-95bf-419a-aa17-40ad048da7bf | None        | None              |            | None                                 |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+

  #temp_sec security group, Works(No downtime)
  [root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp        | 0.0.0.0/0         |            | None                                 |
  | 313ab987-6edc-4514-8854-07a5617c2969 | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
  | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None        | None              |            | None                                 |
  | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
  | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None        | None              |            | None                                 |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+


  When I migrate an instance with default sec group attached to a
  compute it's unavailable untill all the ports are retrieved with bulk
  query (timestamp reachable again: 13:00:03.783):

  #########I've added a log rule for the filter and resource type and query_id#########
  2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69
  2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
  -packages/neutron/agent/resource_cache.py:70
  2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48
  70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

  <snip>

  2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options=
  [],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id
  =8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set(
  [2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

  2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
  tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
  2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413'])

  2018-11-01 12:58:51.447 291758 DEBUG
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
  0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f
  19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
  for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update
  /usr/lib/python2.7/site-
  packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i
  d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e
  :08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po
  licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

  2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
  tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

  2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
  -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

  2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15
  1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i
  d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e
  :0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199

  2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager
  [req-08ac0090-52de-4dde-80f7-90306c250535
  2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - -
  -] Notify callbacks
  ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
  --9223372036854210276',
  'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387']
  for Port, after_update _notify_loop /usr/lib/python2.7/site-
  packages/neutron_lib/callbacks/manager.py:167

  2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
  -0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])

  2018-11-01 12:58:54.804 291758 DEBUG
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
  08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f
  19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
  for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update
  /usr/lib/python2.7/site-
  packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 00030037-fe90-40f3-8022-7ae513303f29
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 001ca232-7826-45bd-94b8-3258db2ca789
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 003e144f-368e-487c-9289-41120eb2a48f
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 00437e10-6633-4a19-92b6-f8e46d051980
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
  resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  ######################## 
  Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs
  ########################

  2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache
  [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources
  returned for queries set([('Port', ('security_group_ids',
  ('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query
  /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92


  #########################
  When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating:
  #########################

  2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter:
  {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:69

  2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
  -packages/neutron/agent/resource_cache.py:70

  2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  query_id: set([('SecurityGroupRule', ('security_group_id', (u
  '13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query
  /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

  2018-11-01 12:54:44.130 290907 DEBUG
  neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c-
  94c4a7f7476e - - - - -]
  neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
  bulk_pull called with arguments (<neutron_lib.context.Context object
  at 0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs':
  {'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}}
  wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66

  2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 1831db02-7538-42ec-b04b-4c8c3d06dadb
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 1c49d1ad-a6f1-470b-bd99-36f32932a6e1
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 3c1d82af-acd7-4543-8922-a96c0c67db67
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 452928bf-d5eb-41ab-b520-f34c33c63058
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 4e9201db-3583-42c7-9c79-372a3394a159
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 4ecc1fa6-d090-4d11-9a3d-323e1f012d40
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 65264a84-ae99-4c90-8626-6dbac869d15d
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: 677d7b02-8943-4824-8fa5-990a02503f03
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: d435b0d1-67bb-4834-9518-5bdf8f843610
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
  [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
  resource ID: fc12d72c-94c9-432b-8f14-79412419c7db
  _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:88

  2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u
  '13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

  #########It is returning just 11 resources which is correct
  value.###############

  
  ######### 
  I can break the working sec group by adding a security group rule with any protocol with a remote-group defined:
  #########

  [root@os-admin (openstack_admin)]# openstack security group rule
  create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol
  any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b-
  46f8-ac30-f13e48f9d4b1

  #Remote group is added:
  [root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | ID                                   | IP Protocol | IP Range          | Port Range | Remote Security Group                |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+
  | 07c52171-065a-4553-b6fb-e43a607bd517 | icmp        | 0.0.0.0/0         |            | None                                 |
  | 313ab987-6edc-4514-8854-07a5617c2969 | tcp         | 192.168.99.220/29 | 22:22      | None                                 |
  | 374a4917-28a4-478c-b4a0-b3a20eb9979a | None        | None              |            | None                                 |
  | 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp         | 192.168.99.220/29 | 8000:8000  | None                                 |
  | b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None        | None              |            | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
  | f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None        | None              |            | None                                 |
  +--------------------------------------+-------------+-------------------+------------+--------------------------------------+

  
  2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69

  2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache
  [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG:
  resourcetype: Port _flood_cache_for_query /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:70

  2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache
  [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG:
  query_id: set([('Port', ('security_group_ids', ('5958c3ad-5d2b-
  46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query
  /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72

  2018-11-01 14:30:24.242 311475 DEBUG
  neutron.api.rpc.handlers.resources_rpc [req-
  24595a6d-9784-4676-b825-e4e41150c0ec - - - - -]
  neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
  bulk_pull called with arguments (<neutron_lib.context.Context object
  at 0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids':
  ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7
  /site-packages/oslo_log/helpers.py:66

  2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache
  [req-643cbd7e-06e9-4576-9e1e-086b86c1726d
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Received resource Port update without any changes: eb9288fe-9c78
  -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
  /site-packages/neutron/agent/resource_cache.py:189

  2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache
  [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port
  eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number
  826->828). Old fields: {'status': u'DOWN', 'binding':
  PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={"migrating_to":
  "compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
  'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding':
  PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={"migrating_to":
  "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
  true, "datapath_type": "system", "ovs_hybrid_plug":
  true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
  [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
  =eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
  66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:197

  2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager
  [req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks
  ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
  'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
  --9223372036854209732'] for Port, after_update _notify_loop
  /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

  2018-11-01 14:30:24.956 311475 DEBUG
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
  dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message
  processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
  /usr/lib/python2.7/site-
  packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache
  [req-643cbd7e-06e9-4576-9e1e-086b86c1726d
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Received resource Port update without any changes: eb9288fe-9c78
  -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
  /site-packages/neutron/agent/resource_cache.py:189

  2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache
  [req-347c971e-3e22-4805-9b95-9a57d75f98aa
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Received resource Port update without any changes: eb9288fe-9c78
  -447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
  /site-packages/neutron/agent/resource_cache.py:189

  2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache
  [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port
  eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number
  829->830). Old fields: {'status': u'ACTIVE'} New fields: {'status':
  u'DOWN'} record_resource_update /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:197

  2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager
  [req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks
  ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
  'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
  --9223372036854209732'] for Port, after_update _notify_loop
  /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

  2018-11-01 14:30:28.381 311475 DEBUG
  neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
  d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message
  processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
  /usr/lib/python2.7/site-
  packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache
  [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated
  (revision_number 830->831). Old fields: {'binding':
  PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={"migrating_to":
  "compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
  true, "datapath_type": "system", "ovs_hybrid_plug":
  true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
  [PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
  =eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
  66cc08eb78f8))]} New fields: {'binding':
  PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
  'binding_levels': []} record_resource_update /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:197

  2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
  2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419

  2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache
  [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated
  (revision_number 831->832). Old fields: {'binding':
  PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
  'binding_levels': []} New fields: {'binding':
  PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter":
  true, "datapath_type": "system", "ovs_hybrid_plug":
  true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
  [PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id
  =eb9288fe-9c78-447c-
  ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
  66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
  packages/neutron/agent/resource_cache.py:197

  2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager
  [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
  a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
  -] Notify callbacks
  ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
  'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
  --9223372036854209732'] for Port, after_update _notify_loop
  /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167

  2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
  2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
  2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G
  <snip>

  #######
  7060 resources matched!:
  #######
  2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92

  2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
  [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
  group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b']
  _apply_port_filter /usr/lib/python2.7/site-
  packages/neutron/agent/securitygroups_rpc.py:135

  2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
  [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
  group information _update_security_group_info /usr/lib/python2.7/site-
  packages/neutron/agent/securitygroups_rpc.py:149

  
  I'm not sure why this behave this way, can someone help me with this issue?

  Regards,
  Paul Peereboom
  ODC-Noord

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1801104/+subscriptions
Follow ups

[Bug 1801104] Re: Migration causes downtime while doing bulk_pull
From: Paul Peereboom, 2018-11-26