yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #75941
[Bug 1801104] Re: Migration causes downtime while doing bulk_pull
After upgrading this site to Queens I cannot reproduce the issue anymore.
Now a migration returns just one resource and there is no timeout when doing a migration:
2018-11-26 10:23:54.312 326318 DEBUG neutron.agent.resource_cache [req-8e2ea784-51f5-49de-ad31-cdcf978d72e3 - - - - -] 1 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b
-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:85
I'll close this issue.
Thanks for your help!
** Changed in: neutron
Status: Confirmed => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1801104
Title:
Migration causes downtime while doing bulk_pull
Status in neutron:
Invalid
Bug description:
I'm trying to figure out why some live-migrations in our environment
causes unreachability of the instance for about ~3 minutes.
When setting neutron-openvswitch in debug mode we can see that the
bulk_pull of resources takes about 2-3 minutes before retrieving about
7057 ports(!). While doing this query the instance is not reachable.
After the instance is migrated to the compute and is pingable again
after 3 minutes we can successfully migrate the instance back and
forth without downtime on the compute node. It looks like the cache is
already filled the second migration and a bulk_pull is not required
again. After some time it looks like the cache needs to be refreshed
and a live-migration causes the instance to be unreachable for minutes
again. We can clear the cache aswell when doing a restart of neutron-
openvswitch-agent.
I was able to reproduce this by adding a security group rule with any
protocol with a remote-group defined and attach that sec group to an
instance. Than migrate the instance to a compute node where neutron-
openvswitch is restarted and no other instance is running on it.
Info and logs:
Openstack version: Pike
OS: Redhat
instanceid: 69fd51c3-d5a6-4ce3-94a0-0e66d62634b6
port_id: eb9288fe-9c78-447c-ba77-e2cfd8df624b
Default sec group id: 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 #This sec group causes downtime on migration
Non-default sec group id: 13f878ef-eba7-43fa-9278-19d599bf8f38 #This sec group works as expected
[root@os-admin (openstack_admin)]# openstack security group show 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2017-10-11T13:13:48Z |
| description | Default security group |
| id | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| name | default |
| project_id | b9bd8eac1587429aa07ae00661f2a07b |
| revision_number | 16 | |
| updated_at | 2018-04-17T08:14:55Z |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@os-admin (openstack_admin)]# openstack security group show 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2018-11-01T12:25:10Z |
| description | |
| id | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
| location | None |
| name | temp_sec |
| project_id | b9bd8eac1587429aa07ae00661f2a07b |
| revision_number | 10 | |
| tags | [] |
| updated_at | 2018-11-01T13:22:21Z |
+-----------------+-----------------------------------------------------+
#Default security group, Broken(downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 08d7e5c8-03aa-4870-86b0-af5b2db00ac3
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 366a55ae-5086-4baf-a55c-bfef0cc38081 | None | None | | None |
| 3e0267fc-3da2-4b9f-a3a0-5facaa1d0819 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| 75ee9c87-1969-421e-bf71-c9244ff1fb49 | icmp | 0.0.0.0/0 | | None |
| b54aaa42-a94a-4e72-9ccd-df8fa691ffb9 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| bffaca24-1a9a-4404-8c21-4ab9d1ed6f46 | None | None | | 08d7e5c8-03aa-4870-86b0-af5b2db00ac3 |
| e8cf4dd5-7489-46c4-92f1-970895914f3d | tcp | 192.168.99.220/29 | 22:22 | None |
| fdfad4d8-95bf-419a-aa17-40ad048da7bf | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
#temp_sec security group, Works(No downtime)
[root@os-admin(openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
When I migrate an instance with default sec group attached to a
compute it's unavailable untill all the ports are retrieved with bulk
query (timestamp reachable again: 13:00:03.783):
#########I've added a log rule for the filter and resource type and query_id#########
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG1: filter: {'security_group_id': (u'08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69
2018-11-01 12:57:51.950 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG2: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70
2018-11-01 12:57:51.951 291758 DEBUG neutron.agent.resource_cache [req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG3: query_id: set([('SecurityGroupRule', ('security_group_id', (u'08d7e5c8-03aa-48
70-86b0-af5b2db00ac3',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72
<snip>
2018-11-01 12:58:51.445 291758 DEBUG neutron.agent.resource_cache [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[],binding=PortBinding,binding_levels=[],created_at=2018-11-01T11:58:50Z,data_plane_status=<?>,description='',device_id='',device_owner='',dhcp_options=
[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=1edbf975-96a1-460f-9576-0bb692615f87,mac_address=fa:16:3e:95:98:90,name='lb-vip-cluster_sub.dev.nl',network_id
=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=3,security=PortSecurity(1edbf975-96a1-460f-9576-0bb692615f87),security_group_ids=set(
[2dd84141-5558-44d8-a2f4-05d998b2c413]),status='DOWN',updated_at=2018-11-01T11:58:50Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199
2018-11-01 12:58:51.446 291758 DEBUG neutron_lib.callbacks.manager [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 12:58:51.446 291758 INFO neutron.agent.securitygroups_rpc [req-0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'2dd84141-5558-44d8-a2f4-05d998b2c413'])
2018-11-01 12:58:51.447 291758 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
0d37fa01-38d7-4d49-b903-3236da162eda 2ab89190bc8d40ecb4ee1f8251ebb54f
19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
for port 1edbf975-96a1-460f-9576-0bb692615f87 port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 12:58:54.767 291758 DEBUG neutron.agent.resource_cache [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:19Z,data_plane_status=<?>,description='',device_i
d='e4f76e1c-ddff-414a-a108-fdef9a040efd',device_owner='compute:gn2a',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=f46d5036-50a4-4742-9808-326cffb09272,mac_address=fa:16:3e
:08:29:1f,name='lb-node-02.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_po
licy_id=None,revision_number=8,security=PortSecurity(f46d5036-50a4-4742-9808-326cffb09272),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199
2018-11-01 12:58:54.768 291758 DEBUG neutron_lib.callbacks.manager [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Notify callbacks ['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854210276', 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_no
tifier-581387'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 12:58:54.768 291758 INFO neutron.agent.securitygroups_rpc [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])
2018-11-01 12:58:54.769 291758 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-10f2f394-59d9-4491-aecc-a63a61c8604f 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd15
1350 - - -] port_update message processed for port f46d5036-50a4-4742-9808-326cffb09272 port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 12:58:54.803 291758 DEBUG neutron.agent.resource_cache [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Received new resource Port: Port(admin_state_up=True,allowed_address_pairs=[AllowedAddressPair],binding=PortBinding,binding_levels=[PortBindingLevel],created_at=2018-11-01T11:42:18Z,data_plane_status=<?>,description='',device_i
d='00e872df-8cd0-4bd4-92a5-f5813ce79a26',device_owner='compute:gn2c',dhcp_options=[],distributed_binding=None,dns=None,fixed_ips=[IPAllocation],id=b11f828e-21ed-4628-a9f2-2d9e8e2f272b,mac_address=fa:16:3e
:0c:56:9c,name='lb-node-01.patrickdev.dev.nl',network_id=8940e979-a2a7-4261-8364-19eac132e1f2,project_id='19e0b25d08d94996b638d293dd151350',qos_policy_id=None,revision_number=8,security=PortSecurity(b11f828e-21ed-4628-a9f2-2d9e8e2f272b),security_group_ids=set([f2fe38fb-51db-4f09-8183-e8983b651888,7ce93b5b-82e2-4c2d-908e-7cf8eba7d109,60632b7d-fd52-4ec6-aa3e-d6187af141e7,dc820fa2-1bca-4089-a48f-6be58e20de34,636c41d9-6fd5-4b1c-96d2-0ceb266fe38e,826252f5-9794-4041-ad48-8a4571fc022f]),status='ACTIVE',updated_at=2018-11-01T11:58:54Z) record_resource_update /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:199
2018-11-01 12:58:54.804 291758 DEBUG neutron_lib.callbacks.manager
[req-08ac0090-52de-4dde-80f7-90306c250535
2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - -
-] Notify callbacks
['neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854210276',
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-581387']
for Port, after_update _notify_loop /usr/lib/python2.7/site-
packages/neutron_lib/callbacks/manager.py:167
2018-11-01 12:58:54.804 291758 INFO neutron.agent.securitygroups_rpc [req-08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f 19e0b25d08d94996b638d293dd151350 - - -] Security group member updated set([u'f2fe38fb-51db-4f09-8183-e8983b651888', u'7ce93b5b-82e2-4c2d-908e-7cf8eba7d109', u'60632b7d-fd52-4ec6-aa3e-d6187af141e7', u'dc820fa2-1bca-4089-a48f-6be58e20de34', u'636c41d9-6fd5-4b1c-96d2
-0ceb266fe38e', u'826252f5-9794-4041-ad48-8a4571fc022f'])
2018-11-01 12:58:54.804 291758 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
08ac0090-52de-4dde-80f7-90306c250535 2ab89190bc8d40ecb4ee1f8251ebb54f
19e0b25d08d94996b638d293dd151350 - - -] port_update message processed
for port b11f828e-21ed-4628-a9f2-2d9e8e2f272b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 13:00:02.312 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 00030037-fe90-40f3-8022-7ae513303f29
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.313 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 001ca232-7826-45bd-94b8-3258db2ca789
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 003e144f-368e-487c-9289-41120eb2a48f
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.314 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 00437e10-6633-4a19-92b6-f8e46d051980
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 13:00:02.315 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] ODCN_LOG:
resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
########################
Apparently all our ports in our environment is matched (total of 7057). I've snipped the rest of the resourceid's from the logs
########################
2018-11-01 13:00:03.783 291758 DEBUG neutron.agent.resource_cache
[req-6180bbc7-ac74-4ebe-b284-f7e2100fcc7c - - - - -] 7057 resources
returned for queries set([('Port', ('security_group_ids',
('08d7e5c8-03aa-4870-86b0-af5b2db00ac3',)))]) _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92
#########################
When I detach the default security group and attach a other security group ( 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 ) to the instance I experience no downtime for the instance while migrating:
#########################
2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: filter:
{'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:69
2018-11-01 12:54:44.129 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG: resourcetype: SecurityGroupRule _flood_cache_for_query /usr/lib/python2.7/site
-packages/neutron/agent/resource_cache.py:70
2018-11-01 12:54:44.130 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
query_id: set([('SecurityGroupRule', ('security_group_id', (u
'13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72
2018-11-01 12:54:44.130 290907 DEBUG
neutron.api.rpc.handlers.resources_rpc [req-c2da227d-0fa7-4157-933c-
94c4a7f7476e - - - - -]
neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
bulk_pull called with arguments (<neutron_lib.context.Context object
at 0x7f8d06eab3d0>, 'SecurityGroupRule') {'filter_kwargs':
{'security_group_id': (u'13f878ef-eba7-43fa-9278-19d599bf8f38',)}}
wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 0fb3d0b8-6b0b-4ae5-93c0-269d6a3f5390
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.162 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 1831db02-7538-42ec-b04b-4c8c3d06dadb
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 1c49d1ad-a6f1-470b-bd99-36f32932a6e1
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 3c1d82af-acd7-4543-8922-a96c0c67db67
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 452928bf-d5eb-41ab-b520-f34c33c63058
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.163 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 4e9201db-3583-42c7-9c79-372a3394a159
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 4ecc1fa6-d090-4d11-9a3d-323e1f012d40
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 65264a84-ae99-4c90-8626-6dbac869d15d
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: 677d7b02-8943-4824-8fa5-990a02503f03
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: d435b0d1-67bb-4834-9518-5bdf8f843610
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.164 290907 DEBUG neutron.agent.resource_cache
[req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] ODCN_LOG:
resource ID: fc12d72c-94c9-432b-8f14-79412419c7db
_flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:88
2018-11-01 12:54:44.165 290907 DEBUG neutron.agent.resource_cache [req-c2da227d-0fa7-4157-933c-94c4a7f7476e - - - - -] 11 resources returned for queries set([('SecurityGroupRule', ('security_group_id', (u
'13f878ef-eba7-43fa-9278-19d599bf8f38',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92
#########It is returning just 11 resources which is correct
value.###############
#########
I can break the working sec group by adding a security group rule with any protocol with a remote-group defined:
#########
[root@os-admin (openstack_admin)]# openstack security group rule
create --remote-group 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 --protocol
any --project b9bd8eac1587429aa07ae00661f2a07b 5958c3ad-5d2b-
46f8-ac30-f13e48f9d4b1
#Remote group is added:
[root@os-admin (openstack_admin)]# openstack security group rule list 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
| 07c52171-065a-4553-b6fb-e43a607bd517 | icmp | 0.0.0.0/0 | | None |
| 313ab987-6edc-4514-8854-07a5617c2969 | tcp | 192.168.99.220/29 | 22:22 | None |
| 374a4917-28a4-478c-b4a0-b3a20eb9979a | None | None | | None |
| 3d97dd93-e359-4295-bc71-8c3e64f66585 | tcp | 192.168.99.220/29 | 8000:8000 | None |
| b75497c6-4df2-43cf-b2f9-9a4c04e5e1b3 | None | None | | 5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1 |
| f374e5fb-06ec-4aae-b4d2-abc0cae321a0 | None | None | | None |
+--------------------------------------+-------------+-------------------+------------+--------------------------------------+
2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: filter: {'security_group_ids': ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)} _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:69
2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG:
resourcetype: Port _flood_cache_for_query /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:70
2018-11-01 14:30:24.242 311475 DEBUG neutron.agent.resource_cache
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG:
query_id: set([('Port', ('security_group_ids', ('5958c3ad-5d2b-
46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query
/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:72
2018-11-01 14:30:24.242 311475 DEBUG
neutron.api.rpc.handlers.resources_rpc [req-
24595a6d-9784-4676-b825-e4e41150c0ec - - - - -]
neutron.api.rpc.handlers.resources_rpc.ResourcesPullRpcApi method
bulk_pull called with arguments (<neutron_lib.context.Context object
at 0x7f6dd42d0790>, 'Port') {'filter_kwargs': {'security_group_ids':
('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)}} wrapper /usr/lib/python2.7
/site-packages/oslo_log/helpers.py:66
2018-11-01 14:30:24.531 311475 DEBUG neutron.agent.resource_cache
[req-643cbd7e-06e9-4576-9e1e-086b86c1726d
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Received resource Port update without any changes: eb9288fe-9c78
-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
/site-packages/neutron/agent/resource_cache.py:189
2018-11-01 14:30:24.955 311475 DEBUG neutron.agent.resource_cache
[req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Resource Port
eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number
826->828). Old fields: {'status': u'DOWN', 'binding':
PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} New fields: {'status': u'ACTIVE', 'binding':
PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197
2018-11-01 14:30:24.956 311475 DEBUG neutron_lib.callbacks.manager
[req-dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:24.956 311475 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
dea67ef8-eb0e-411e-9dfe-2d547e6f9ec8 - - - - -] port_update message
processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:30:25.124 311475 DEBUG neutron.agent.resource_cache
[req-643cbd7e-06e9-4576-9e1e-086b86c1726d
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Received resource Port update without any changes: eb9288fe-9c78
-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
/site-packages/neutron/agent/resource_cache.py:189
2018-11-01 14:30:28.266 311475 DEBUG neutron.agent.resource_cache
[req-347c971e-3e22-4805-9b95-9a57d75f98aa
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Received resource Port update without any changes: eb9288fe-9c78
-447c-ba77-e2cfd8df624b record_resource_update /usr/lib/python2.7
/site-packages/neutron/agent/resource_cache.py:189
2018-11-01 14:30:28.380 311475 DEBUG neutron.agent.resource_cache
[req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Resource Port
eb9288fe-9c78-447c-ba77-e2cfd8df624b updated (revision_number
829->830). Old fields: {'status': u'ACTIVE'} New fields: {'status':
u'DOWN'} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197
2018-11-01 14:30:28.381 311475 DEBUG neutron_lib.callbacks.manager
[req-d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:28.381 311475 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
d1ddf17b-7b19-4147-b812-968079eb226f - - - - -] port_update message
processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update
/usr/lib/python2.7/site-
packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:30:29.336 311475 DEBUG neutron.agent.resource_cache
[req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated
(revision_number 830->831). Old fields: {'binding':
PortBinding(host='compute2.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={"migrating_to":
"compute1.odcnoord.nl"},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute2.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} New fields: {'binding':
PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197
2018-11-01 14:30:29.337 311475 DEBUG neutron_lib.callbacks.manager [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] Notify callbacks ['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479', 'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update--9223372036854209732'] for Port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:29.337 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:30:30.007 311475 DEBUG neutron.agent.resource_cache
[req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Resource Port eb9288fe-9c78-447c-ba77-e2cfd8df624b updated
(revision_number 831->832). Old fields: {'binding':
PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal'),
'binding_levels': []} New fields: {'binding':
PortBinding(host='compute1.odcnoord.nl',port_id=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,profile={},status='ACTIVE',vif_details={"port_filter":
true, "datapath_type": "system", "ovs_hybrid_plug":
true},vif_type='ovs',vnic_type='normal'), 'binding_levels':
[PortBindingLevel(driver='openvswitch',host='compute1.odcnoord.nl',level=0,port_id
=eb9288fe-9c78-447c-
ba77-e2cfd8df624b,segment=NetworkSegment(a9661f45-a9a5-4b8b-9e3e-
66cc08eb78f8))]} record_resource_update /usr/lib/python2.7/site-
packages/neutron/agent/resource_cache.py:197
2018-11-01 14:30:30.008 311475 DEBUG neutron_lib.callbacks.manager
[req-94f4886e-be2d-45ad-a05b-68bacaf8e06f
a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - -
-] Notify callbacks
['neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-460479',
'neutron.api.rpc.handlers.securitygroups_rpc.SecurityGroupServerAPIShim._handle_sg_member_update
--9223372036854209732'] for Port, after_update _notify_loop
/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-11-01 14:30:30.008 311475 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-94f4886e-be2d-45ad-a05b-68bacaf8e06f a192a9f466b940e2af8e18ac477a8643 4c59ee25d56a4455b975ab17cd0bd351 - - -] port_update message processed for port eb9288fe-9c78-447c-ba77-e2cfd8df624b port_update /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:419
2018-11-01 14:32:37.289 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00030037-fe90-40f3-8022-7ae513303f29 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.290 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00138bf7-aa0e-41fc-bd43-00d0f33e56b3 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 001ca232-7826-45bd-94b8-3258db2ca789 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 002dd5f1-7ceb-45b6-b54a-f3ac3b8c6402 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003e144f-368e-487c-9289-41120eb2a48f _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.291 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 003efeb1-fdbc-4a88-90df-b2471159fa7c _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 00437e10-6633-4a19-92b6-f8e46d051980 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 004399f4-6340-4cb3-9da0-9ac55cd14e50 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88
2018-11-01 14:32:37.292 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] ODCN_LOG: resource ID: 0045bf29-66a6-4277-bfbf-ccc3ff4744e7 _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:88G
<snip>
#######
7060 resources matched!:
#######
2018-11-01 14:32:38.954 311475 DEBUG neutron.agent.resource_cache [req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] 7060 resources returned for queries set([('Port', ('security_group_ids', ('5958c3ad-5d2b-46f8-ac30-f13e48f9d4b1',)))]) _flood_cache_for_query /usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py:92
2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
group information for ports [u'eb9288fe-9c78-447c-ba77-e2cfd8df624b']
_apply_port_filter /usr/lib/python2.7/site-
packages/neutron/agent/securitygroups_rpc.py:135
2018-11-01 14:32:38.971 311475 DEBUG neutron.agent.securitygroups_rpc
[req-24595a6d-9784-4676-b825-e4e41150c0ec - - - - -] Update security
group information _update_security_group_info /usr/lib/python2.7/site-
packages/neutron/agent/securitygroups_rpc.py:149
I'm not sure why this behave this way, can someone help me with this issue?
Regards,
Paul Peereboom
ODC-Noord
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1801104/+subscriptions
References
