yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #81498
[Bug 1861496] Re: All ports of server instance are open even no security group does allow this
Hi Sebastian,
To avoid this problem You can remove from Your SG rules with
remote_security_group_id and only use CIDRs in rules. Than there
shouldn't be added IPs from other ports which are using same security
group in ipset.
I will mark this bug as "Invalid" now.
** Changed in: neutron
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1861496
Title:
All ports of server instance are open even no security group does
allow this
Status in neutron:
Invalid
Bug description:
-------------------------
System environment
-------------------------
Openstack: Queens
Ubuntu: 18.04.3 LTS
Neutron: 12.1.0
Nova: 17.0.12
Neutron is configured to have DVR and HA router.
-------------------------
Problem description
-------------------------
We have several projects in Openstack, where the default security gropu created during the initial setup of the project does now open all ports for instances, which have a floating ip assigned. Our expected behaviour is that all ports are closed for public access through the floating ip. For all other projects in the same Openstack installation, this is the case. Also if we use a new security group with identical rules as the default security group we cannot reproduce this error. Creating new security groups does fix the issue as a workaround. But still we are not sure, that there is no general issue for the future.
Security Group Rules
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2019-08-19T17:18:29Z |
| description | Default security group |
| id | 77015487-2991-4924-afd8-7b9468cacd3e |
| location | Munch({'project': Munch({'domain_id': None, 'id': u'793d7760e0da415ca14013e5aaa0fb36', 'name': None, 'domain_name': None}), 'cloud': '', 'region_name': 'RegionOne', 'zone': None}) |
| name | default |
| project_id | 793d7760e0da415ca14013e5aaa0fb36 |
| revision_number | 27 |
| rules | created_at='2019-11-22T07:57:32Z', direction='ingress', ethertype='IPv4', id='2a579604-50c6-4bd9-9ec7-07d835adca9d', remote_group_id='77015487-2991-4924-afd8-7b9468cacd3e', updated_at='2019-11-22T07:57:32Z' |
| | created_at='2019-11-22T07:59:19Z', direction='egress', ethertype='IPv6', id='9cbb5404-4ecc-46ef-b582-39895a7cdf2a', updated_at='2019-11-22T07:59:19Z' |
| | created_at='2019-11-22T07:57:51Z', direction='ingress', ethertype='IPv6', id='afd436a5-273a-479e-9829-a62da3df7e0a', remote_group_id='77015487-2991-4924-afd8-7b9468cacd3e', updated_at='2019-11-22T07:57:51Z' |
| | created_at='2019-08-20T08:40:14Z', direction='ingress', ethertype='IPv4', id='de9dbe59-7c2a-4f20-8281-b62a41416538', protocol='icmp', remote_ip_prefix='0.0.0.0/0', updated_at='2019-08-20T08:40:14Z' |
| | created_at='2019-08-19T17:18:29Z', direction='egress', ethertype='IPv4', id='fa3e699e-b8ce-4a79-bddb-db19233214e5', updated_at='2019-08-19T17:18:29Z' |
| tags | [] |
| updated_at | 2019-11-22T07:59:19Z |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-------------------------
Investigation (so far)
-------------------------
We have checked the iptables entries for a server instance showing the described behaviour on the cpu node. We were not able to identify any issue compared to other instances in other projects not having the issue. We also checked the iptables rules of the qrouter responsible for the floating ip.
We found the following relevant rules for the instance:
-A neutron-openvswi-INPUT -m physdev --physdev-in tap68c54c8c-33 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-o68c54c8c-3
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap68c54c8c-33
--physdev-is-bridged -m comment --comment "Jump to the VM specific
chain." -j neutron-openvswi-o68c54c8c-3
-A neutron-openvswi-o68c54c8c-3 -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN
-A neutron-openvswi-o68c54c8c-3 -j neutron-openvswi-s68c54c8c-3
-A neutron-openvswi-s68c54c8c-3 -s 10.0.5.11/32 -m mac --mac-source FA:16:3E:41:5A:96 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-s68c54c8c-3 -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP
-A neutron-openvswi-o68c54c8c-3 -p udp -m udp --sport 68 --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN
-A neutron-openvswi-o68c54c8c-3 -p udp -m udp --sport 67 --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP
-A neutron-openvswi-o68c54c8c-3 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-o68c54c8c-3 -j RETURN
-A neutron-openvswi-o68c54c8c-3 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-o68c54c8c-3 -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-sg-chain -j ACCEPT
Further we tried to add a rule to the new security group referencing to remote_group = default like: direction=ingress, protocoll=any, ethertype=IPv4 and remote_group_id=<id_of_default>. We wanted to test, what happens, if we try to allow the access to the machine for all instances belonging to the default security group as well. As soon as we added this rule, we could reach every port for the test instance via the floating ip again. After deleting the rule referencing to the default security group, the ports were not reachable anymore.
After we created new security groups and assigned them to the
instances, like the default security group before, we hat the same
issue for the newly created security groups. Even after removing them
and only reassigning to one single server, all ports for this server
are open.
So we have projects, which do not show this behaviour and we have
projects, which do. It is independent from the CPU node and can be
reproduced on different ones, even with new server instances or
security groups.
-------------------------
Further investigation
-------------------------
We would like to investigate further into this issue, to find out, why this behaviour happens and if it is a bug or only a misconfiguration. Can someone please suggest what information to provide in addition to nail down this issue?
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1861496/+subscriptions
References
