yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #93471
[Bug 2028159] Re: Invalid IPv6 subnet in self-service network breaks DHCP agent
Reviewed: https://review.opendev.org/c/openstack/neutron/+/905148
Committed: https://opendev.org/openstack/neutron/commit/2f0011194012a2482f79603c310028736e9ff3c8
Submitter: "Zuul (22348)"
Branch: master
commit 2f0011194012a2482f79603c310028736e9ff3c8
Author: Brian Haley <haleyb.dev@xxxxxxxxx>
Date: Mon Jan 8 15:50:40 2024 -0500
Disallow subnet cidr of :: without PD
Do not allow the subnet cidr of :: to be used when
creating a subnet, except in the case IPv6 prefix
delegation has been specified in the request.
Closes-bug: #2028159
Change-Id: I480e9a117513996f3c070acd4ba39c2b9fe9c0f1
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2028159
Title:
Invalid IPv6 subnet in self-service network breaks DHCP agent
Status in neutron:
Fix Released
Status in OpenStack Security Advisory:
Incomplete
Bug description:
High level description:
A user creates self-service network (vxlan) with IPv6 subnet with the address ::/24, gateway ::
After that new instances in other networks do not receive addresses via DHCP.
Pre-conditions:
Neutron 20.3.1 (Yoga) with OVS ML2 plugin
3 DHCP agents for each network running on each of 3 controllers
A user account with a user role in some project
Step-by-step reproduction steps:
1. Launch a new instance in any DHCP-enabled network.
2. Verify that the instance receives an address.
3. Create a new network with a subnet with the following options:
a) via Dashboard:
Network Address: ::/24
IP Version: IPv6
Gateway IP: ::
Enable DHCP: true
IPv6 Address Configuration Mode: No options specified
b) or via CLI:
openstack network create bad
openstack subnet create --network bad --dhcp --ip-version 6 --subnet-range "::/24" --gateway "::" badsub
4. Launch another instance in the same network as #1.
5. Verify that the instance does not receive an address.
6. Delete the network from step 3.
7. Reboot the last instance.
8. Verify that it receives an address.
Expected output:
Either Neutron does not allow to create such subnet, or
New instances do receive addresses (DHCP agent stays uninterrupted)
Actual output:
Neutron did not perform verification of the subnet options.
DHCP agent enters a broken state, new instances do not receive addresses.
Version:
# rpm -qa | grep neutron | sort
openstack-neutron-20.3.1-1.el8.noarch
openstack-neutron-common-20.3.1-1.el8.noarch
openstack-neutron-ml2-20.3.1-1.el8.noarch
openstack-neutron-openvswitch-20.3.1-1.el8.noarch
python3-neutron-20.3.1-1.el8.noarch
python3-neutronclient-7.8.0-1.el8.noarch
python3-neutron-lib-2.20.2-1.el8.noarch
# cat /etc/redhat-release
CentOS Stream release 8
# uname -srvmpio
Linux 4.18.0-383.el8.x86_64 #1 SMP Wed Apr 20 15:38:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Environment:
# openstack compute service list --sort-column Host
+--------------------------------------+----------------+------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+--------------------------------------+----------------+------+----------+---------+-------+----------------------------+
| c45e81ed-e173-4e36-b209-01c80b99036d | nova-conductor | s5 | internal | enabled | up | 2023-07-19T12:05:47.000000 |
| c0310488-c0c5-4c37-9847-44259c86f776 | nova-scheduler | s5 | internal | enabled | up | 2023-07-19T12:05:48.000000 |
| b30d037e-90c2-4624-b8a0-91822ecf85a8 | nova-conductor | s6 | internal | enabled | up | 2023-07-19T12:05:55.000000 |
| da00e178-c2a5-487c-affa-10ed60cc3a2f | nova-scheduler | s6 | internal | enabled | up | 2023-07-19T12:05:49.000000 |
| 49e63486-c55f-428b-a1a1-defac0f47bb7 | nova-conductor | s7 | internal | enabled | up | 2023-07-19T12:05:53.000000 |
| ae929e33-a114-4446-8c7a-a1f9a8aa5c21 | nova-scheduler | s7 | internal | enabled | up | 2023-07-19T12:05:55.000000 |
| 0e10eb67-8150-4a3d-a268-ec9e1a3cc0ec | nova-compute | s8 | nova | enabled | up | 2023-07-19T12:05:46.000000 |
| d271bf37-4d47-4150-8cd2-7119fcebc1a6 | nova-compute | s9 | nova | enabled | up | 2023-07-19T12:05:54.000000 |
+--------------------------------------+----------------+------+----------+---------+-------+----------------------------+
# openstack network agent list --sort-column Binary --sort-column Host
+--------------------------------------+--------------------+------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+------+-------------------+-------+-------+---------------------------+
| d749fb1b-2bda-42bf-b5a4-dd6a6c0f56c2 | DHCP agent | s5 | nova | :-) | UP | neutron-dhcp-agent |
| cceea512-154c-44ea-a571-9e5a542ccde9 | DHCP agent | s6 | nova | :-) | UP | neutron-dhcp-agent |
| 5c5ad312-c1ab-4d33-9e54-b62e7112b218 | DHCP agent | s7 | nova | :-) | UP | neutron-dhcp-agent |
| 7dc0b55f-6a3c-45bc-866a-28540128147d | L3 agent | s5 | nova | :-) | UP | neutron-l3-agent |
| 6171f6e5-66b6-475a-ba6b-6cc113dd2729 | L3 agent | s6 | nova | :-) | UP | neutron-l3-agent |
| df9b3796-181b-46ab-8adb-52083cbc5d1a | L3 agent | s7 | nova | :-) | UP | neutron-l3-agent |
| 03cffc3b-3e27-48bf-a633-b5ffed011fa6 | L3 agent | s8 | nova | :-) | UP | neutron-l3-agent |
| 1430f493-57e4-436d-8fcb-d8344fdbb2b0 | L3 agent | s9 | nova | :-) | UP | neutron-l3-agent |
| 52bd49c0-96d3-410f-88bb-ea99550851bc | Metadata agent | s5 | None | :-) | UP | neutron-metadata-agent |
| 699aca37-efc3-4c42-ad2c-eb6d5897a203 | Metadata agent | s6 | None | :-) | UP | neutron-metadata-agent |
| 89588d09-93ca-4c92-b544-0fd16274f4c9 | Metadata agent | s7 | None | :-) | UP | neutron-metadata-agent |
| e9af410b-7237-4e25-adcc-c13483917bf4 | Metadata agent | s8 | None | :-) | UP | neutron-metadata-agent |
| b4e9bef5-36fe-4953-a2f9-8d437fe7b30f | Metadata agent | s9 | None | :-) | UP | neutron-metadata-agent |
| 7173b0ed-4ec5-4177-ba29-3782e3e5f8be | Open vSwitch agent | s5 | None | :-) | UP | neutron-openvswitch-agent |
| d58ca721-f56d-4b3a-85d7-5e6c0d04f9db | Open vSwitch agent | s6 | None | :-) | UP | neutron-openvswitch-agent |
| 2924fb03-7e16-42c5-8af8-c1a3b25b0905 | Open vSwitch agent | s7 | None | :-) | UP | neutron-openvswitch-agent |
| b2118af9-a418-469f-9fea-379a92aa8548 | Open vSwitch agent | s8 | None | :-) | UP | neutron-openvswitch-agent |
| ee1c3f12-be03-4891-8895-b8f72f417585 | Open vSwitch agent | s9 | None | :-) | UP | neutron-openvswitch-agent |
+--------------------------------------+--------------------+------+-------------------+-------+-------+---------------------------+
Perceived severity:
High
dhcp-agent.log contains the following:
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for eb2e3a84-87fa-4d03-87fa-8986a70f5d57.: pr2modules.netlink.exceptions.NetlinkError: (99, 'Cannot assign requested address')
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 275, in enable
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent while not predicate():
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 287, in _enable
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1780, in setup
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent namespace=network.namespace)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 152, in init_l3
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent device.addr.add(ip_cidr)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 541, in add
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent add_broadcast)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 830, in add_ip_address
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent device, namespace, scope, broadcast)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent r_call_timeout)
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2])
2023-07-14 16:26:03.589 93091 ERROR neutron.agent.dhcp.agent pr2modules.netlink.exceptions.NetlinkError: (99, 'Cannot assign requested address')
or
2023-07-19 13:58:39.777 98250 DEBUG neutron.agent.linux.dhcp [req-82f865b9-f787-4983-acb2-145c7db53877 - - - - -] Setting IPv6 gateway for dhcp netns on net 94355373-4bb8-4117-bec3-c6f492f26a93 to :: _set_default_route_ip_version /usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py:1464
2023-07-19 13:58:39.832 98645 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[14221983-9b1e-49c3-8248-59325d3e4069]: (22, 'Invalid argument') _process_cmd /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:481
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 476, in _process_cmd
ret = func(*f_args, **f_kwargs)
File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 274, in _wrap
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 752, in add_ip_route
ip.route('replace', **kwargs)
File "/usr/lib/python3.6/site-packages/pr2modules/iproute/linux.py", line 2042, in route
callback=callback)
File "/usr/lib/python3.6/site-packages/pr2modules/netlink/nlsocket.py", line 397, in nlm_request
return tuple(self._genlm_request(*argv, **kwarg))
File "/usr/lib/python3.6/site-packages/pr2modules/netlink/nlsocket.py", line 891, in nlm_request
callback=callback):
File "/usr/lib/python3.6/site-packages/pr2modules/netlink/nlsocket.py", line 400, in get
return tuple(self._genlm_get(*argv, **kwarg))
File "/usr/lib/python3.6/site-packages/pr2modules/netlink/nlsocket.py", line 725, in get
raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (22, 'Invalid argument')
2023-07-19 13:58:39.834 98645 DEBUG oslo.privsep.daemon [-] privsep: reply[14221983-9b1e-49c3-8248-59325d3e4069]: (5, 'pr2modules.netlink.exceptions.NetlinkError', (22, 'Invalid argument')) _call_back /usr/lib/python3.6/site-packages/oslo_privsep/daemon.py:502
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent [req-82f865b9-f787-4983-acb2-145c7db53877 - - - - -] Unable to enable dhcp for 94355373-4bb8-4117-bec3-c6f492f26a93.: pr2modules.netlink.exceptions.NetlinkError: (22, 'Invalid argument')
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent [req-82f865b9-f787-4983-acb2-145c7db53877 - - - - -] Unable to enable dhcp for 94355373-4bb8-4117-bec3-c6f492f26a93.: pr2modules.netlink.exceptions.NetlinkError: (22, 'Invalid argument')
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 218, in call_driver
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 275, in enable
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 717, in wait_until_true
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent while not predicate():
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 287, in _enable
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1782, in setup
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent self._set_default_route(network, interface_name)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1505, in _set_default_route
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent ip_version)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1483, in _set_default_route_ip_version
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent device.route.add_gateway(subnet.gateway_ip)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 620, in add_gateway
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent scope=scope)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 658, in add_route
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent table=table, metric=metric, scope=scope, **kwargs)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 1532, in add_ip_route
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent metric=metric, scope=scope, proto=proto, **kwargs)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent r_call_timeout)
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2])
2023-07-19 13:58:39.837 98250 ERROR neutron.agent.dhcp.agent pr2modules.netlink.exceptions.NetlinkError: (22, 'Invalid argument')
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2028159/+subscriptions
