yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87375
[Bug 1946493] [NEW] Using "set-name" with interface specific DNS with a v2 network config causes a KeyError
Public bug reported:
This bug was first reported at https://github.com/kubernetes-sigs/image-
builder/issues/712 and occurs when the v2 network configuration
directive "set-name" is used in conjunction with interface specific DNS
settings.
Cloud-Provider: VMware, but does not matter as this bug is distro and DS agnostic
Config: The metadata was set to the following:
instance-id: "wlan-1-md-0-775d8846bf-9bfrd"
local-hostname: "wlan-1-md-0-775d8846bf-9bfrd"
wait-on-network:
ipv4: false
ipv6: false
network:
version: 2
ethernets:
id0:
match:
macaddress: "00:50:56:a1:d8:a7"
set-name: "eth0"
wakeonlan: true
addresses:
- "10.196.27.122/28"
gateway4: "10.196.27.126"
nameservers:
addresses:
- "10.102.102.132"
- "10.102.102.133"
- "10.90.24.1"
search:
- "refsa1.bn.schiff.telekom.de"
Again though, the platform is likely irrelevant as this seems to be a
bug introduced with https://github.com/canonical/cloud-
init/commit/abd2da5777195e7e432b0d53a3f7f29d071dd50e, and can occur on
any platform with and datasource as long as network v2 config is used
with "set-name" and interface specific DNS.
The bug can be surfaced via unit test by patching the v21.3 version of
Cloud-Init with the following:
diff --git a/cloudinit/net/tests/test_network_state.py b/cloudinit/net/tests/test_network_state.py
index 84e8308a..c0aa78a0 100644
--- a/cloudinit/net/tests/test_network_state.py
+++ b/cloudinit/net/tests/test_network_state.py
@@ -52,6 +52,7 @@ network:
eth1:
match:
macaddress: '66:77:88:99:00:11'
+ set-name: "eth2"
nameservers:
search: [foo.local, bar.local]
addresses: [4.4.4.4]
Next, run the affected test from the root of the Cloud-Init source tree:
$ make clean_pyc && \
PYTHONPATH="$(pwd)" \
python3 -m pytest -v cloudinit/net/tests/test_network_state.py
The output will resemble the following:
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/akutz/Projects/cloud-init, configfile: tox.ini
collected 10 items
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_empty_v1_config_gets_network_state PASSED [ 10%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_empty_v2_config_gets_network_state PASSED [ 20%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_missing_version_returns_none PASSED [ 30%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_unknown_versions_returns_none PASSED [ 40%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_valid_config_gets_network_state PASSED [ 50%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_version_2_passes_self_as_config PASSED [ 60%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfigV2::test_version_2_ignores_renderer_key PASSED [ 70%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v1_nameservers_valid PASSED [ 80%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v1_nameservers_invalid PASSED [ 90%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v2_nameservers FAILED [100%]
=========================================================================== FAILURES ============================================================================
_____________________________________________________ TestNetworkStateParseNameservers.test_v2_nameservers ______________________________________________________
self = <cloudinit.net.tests.test_network_state.TestNetworkStateParseNameservers object at 0x10a7db9d0>
def test_v2_nameservers(self):
> config = self._parse_network_state_from_config(V2_CONFIG_NAMESERVERS)
cloudinit/net/tests/test_network_state.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cloudinit/net/tests/test_network_state.py:114: in _parse_network_state_from_config
return network_state.parse_net_config_data(yaml['network'])
cloudinit/net/network_state.py:1074: in parse_net_config_data
nsi.parse_config(skip_broken=skip_broken)
cloudinit/net/network_state.py:261: in parse_config
self.parse_config_v2(skip_broken=skip_broken)
cloudinit/net/network_state.py:310: in parse_config_v2
self._v2_common(command)
cloudinit/net/network_state.py:722: in _v2_common
self._handle_individual_nameserver(name_cmd, iface)
cloudinit/net/network_state.py:91: in decorator
return func(self, command, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <cloudinit.net.network_state.NetworkStateInterpreter object at 0x10a7e4f10>
command = {'address': ['4.4.4.4'], 'search': ['foo.local', 'bar.local'], 'type': 'nameserver'}, iface = 'eth1'
@ensure_command_keys(['address'])
def _handle_individual_nameserver(self, command, iface):
_iface = self._network_state.get('interfaces')
nameservers, search = self._parse_dns(command)
> _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
E KeyError: 'eth1'
cloudinit/net/network_state.py:546: KeyError
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:670 v2(ethernets) -> v1(physical):
{'type': 'physical', 'name': 'eth0', 'mac_address': '00:11:22:33:44:55', 'match': {'macaddress': '00:11:22:33:44:55'}}
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:670 v2(ethernets) -> v1(physical):
{'type': 'physical', 'name': 'eth2', 'mac_address': '66:77:88:99:00:11', 'match': {'macaddress': '66:77:88:99:00:11'}}
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:711 v2_common: handling config:
{'eth0': {'match': {'macaddress': '00:11:22:33:44:55'}, 'nameservers': {'search': ['spam.local', 'eggs.local'], 'addresses': ['8.8.8.8']}}, 'eth1': {'match': {'macaddress': '66:77:88:99:00:11'}, 'set-name': 'eth2', 'nameservers': {'search': ['foo.local', 'bar.local'], 'addresses': ['4.4.4.4']}}}
======================================================================= warnings summary ========================================================================
../../../../usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py:1183
/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py:1183: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
self.issue_config_time_warning(
conftest.py:68
/Users/akutz/Projects/cloud-init/conftest.py:68: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
@pytest.yield_fixture(autouse=True)
conftest.py:169
/Users/akutz/Projects/cloud-init/conftest.py:169: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
def httpretty():
-- Docs: https://docs.pytest.org/en/stable/warnings.html
==================================================================== short test summary info ====================================================================
FAILED cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v2_nameservers - KeyError: 'eth1'
============================================================ 1 failed, 9 passed, 3 warnings in 0.60s ============================================================
This is is occurring because the code to iterate over the interfaces
when configuring interface-specific DNS is using the original interface
name, not the one from the "set-name" directive. The following patch
corrects the issue:
diff --git a/cloudinit/net/network_state.py b/cloudinit/net/network_state.py
index 95b064f0..06ff8e96 100644
--- a/cloudinit/net/network_state.py
+++ b/cloudinit/net/network_state.py
@@ -543,7 +543,11 @@ class NetworkStateInterpreter(metaclass=CommandHandlerMeta):
def _handle_individual_nameserver(self, command, iface):
_iface = self._network_state.get('interfaces')
nameservers, search = self._parse_dns(command)
- _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
+ try:
+ _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
+ except:
+ print("original iface name: %s\niface dict: %s\n" % (iface, _iface))
+ raise
@ensure_command_keys(['destination'])
def handle_route(self, command):
diff --git a/cloudinit/net/tests/test_network_state.py b/cloudinit/net/tests/test_network_state.py
index 84e8308a..45e99171 100644
--- a/cloudinit/net/tests/test_network_state.py
+++ b/cloudinit/net/tests/test_network_state.py
@@ -52,6 +52,7 @@ network:
eth1:
match:
macaddress: '66:77:88:99:00:11'
+ set-name: "ens92"
nameservers:
search: [foo.local, bar.local]
addresses: [4.4.4.4]
Now the above test passes. A PR will be opened on Cloud-Init's GitHub
repository with the above patch.
** Affects: cloud-init
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1946493
Title:
Using "set-name" with interface specific DNS with a v2 network config
causes a KeyError
Status in cloud-init:
New
Bug description:
This bug was first reported at https://github.com/kubernetes-
sigs/image-builder/issues/712 and occurs when the v2 network
configuration directive "set-name" is used in conjunction with
interface specific DNS settings.
Cloud-Provider: VMware, but does not matter as this bug is distro and DS agnostic
Config: The metadata was set to the following:
instance-id: "wlan-1-md-0-775d8846bf-9bfrd"
local-hostname: "wlan-1-md-0-775d8846bf-9bfrd"
wait-on-network:
ipv4: false
ipv6: false
network:
version: 2
ethernets:
id0:
match:
macaddress: "00:50:56:a1:d8:a7"
set-name: "eth0"
wakeonlan: true
addresses:
- "10.196.27.122/28"
gateway4: "10.196.27.126"
nameservers:
addresses:
- "10.102.102.132"
- "10.102.102.133"
- "10.90.24.1"
search:
- "refsa1.bn.schiff.telekom.de"
Again though, the platform is likely irrelevant as this seems to be a
bug introduced with https://github.com/canonical/cloud-
init/commit/abd2da5777195e7e432b0d53a3f7f29d071dd50e, and can occur on
any platform with and datasource as long as network v2 config is used
with "set-name" and interface specific DNS.
The bug can be surfaced via unit test by patching the v21.3 version of
Cloud-Init with the following:
diff --git a/cloudinit/net/tests/test_network_state.py b/cloudinit/net/tests/test_network_state.py
index 84e8308a..c0aa78a0 100644
--- a/cloudinit/net/tests/test_network_state.py
+++ b/cloudinit/net/tests/test_network_state.py
@@ -52,6 +52,7 @@ network:
eth1:
match:
macaddress: '66:77:88:99:00:11'
+ set-name: "eth2"
nameservers:
search: [foo.local, bar.local]
addresses: [4.4.4.4]
Next, run the affected test from the root of the Cloud-Init source
tree:
$ make clean_pyc && \
PYTHONPATH="$(pwd)" \
python3 -m pytest -v cloudinit/net/tests/test_network_state.py
The output will resemble the following:
====================================================================== test session starts ======================================================================
platform darwin -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/akutz/Projects/cloud-init, configfile: tox.ini
collected 10 items
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_empty_v1_config_gets_network_state PASSED [ 10%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_empty_v2_config_gets_network_state PASSED [ 20%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_missing_version_returns_none PASSED [ 30%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_unknown_versions_returns_none PASSED [ 40%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_valid_config_gets_network_state PASSED [ 50%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfig::test_version_2_passes_self_as_config PASSED [ 60%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseConfigV2::test_version_2_ignores_renderer_key PASSED [ 70%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v1_nameservers_valid PASSED [ 80%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v1_nameservers_invalid PASSED [ 90%]
cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v2_nameservers FAILED [100%]
=========================================================================== FAILURES ============================================================================
_____________________________________________________ TestNetworkStateParseNameservers.test_v2_nameservers ______________________________________________________
self = <cloudinit.net.tests.test_network_state.TestNetworkStateParseNameservers object at 0x10a7db9d0>
def test_v2_nameservers(self):
> config = self._parse_network_state_from_config(V2_CONFIG_NAMESERVERS)
cloudinit/net/tests/test_network_state.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cloudinit/net/tests/test_network_state.py:114: in _parse_network_state_from_config
return network_state.parse_net_config_data(yaml['network'])
cloudinit/net/network_state.py:1074: in parse_net_config_data
nsi.parse_config(skip_broken=skip_broken)
cloudinit/net/network_state.py:261: in parse_config
self.parse_config_v2(skip_broken=skip_broken)
cloudinit/net/network_state.py:310: in parse_config_v2
self._v2_common(command)
cloudinit/net/network_state.py:722: in _v2_common
self._handle_individual_nameserver(name_cmd, iface)
cloudinit/net/network_state.py:91: in decorator
return func(self, command, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <cloudinit.net.network_state.NetworkStateInterpreter object at 0x10a7e4f10>
command = {'address': ['4.4.4.4'], 'search': ['foo.local', 'bar.local'], 'type': 'nameserver'}, iface = 'eth1'
@ensure_command_keys(['address'])
def _handle_individual_nameserver(self, command, iface):
_iface = self._network_state.get('interfaces')
nameservers, search = self._parse_dns(command)
> _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
E KeyError: 'eth1'
cloudinit/net/network_state.py:546: KeyError
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:670 v2(ethernets) -> v1(physical):
{'type': 'physical', 'name': 'eth0', 'mac_address': '00:11:22:33:44:55', 'match': {'macaddress': '00:11:22:33:44:55'}}
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:670 v2(ethernets) -> v1(physical):
{'type': 'physical', 'name': 'eth2', 'mac_address': '66:77:88:99:00:11', 'match': {'macaddress': '66:77:88:99:00:11'}}
2021-10-08 00:57:41 DEBUG cloudinit.net.network_state:network_state.py:711 v2_common: handling config:
{'eth0': {'match': {'macaddress': '00:11:22:33:44:55'}, 'nameservers': {'search': ['spam.local', 'eggs.local'], 'addresses': ['8.8.8.8']}}, 'eth1': {'match': {'macaddress': '66:77:88:99:00:11'}, 'set-name': 'eth2', 'nameservers': {'search': ['foo.local', 'bar.local'], 'addresses': ['4.4.4.4']}}}
======================================================================= warnings summary ========================================================================
../../../../usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py:1183
/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py:1183: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
self.issue_config_time_warning(
conftest.py:68
/Users/akutz/Projects/cloud-init/conftest.py:68: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
@pytest.yield_fixture(autouse=True)
conftest.py:169
/Users/akutz/Projects/cloud-init/conftest.py:169: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
def httpretty():
-- Docs: https://docs.pytest.org/en/stable/warnings.html
==================================================================== short test summary info ====================================================================
FAILED cloudinit/net/tests/test_network_state.py::TestNetworkStateParseNameservers::test_v2_nameservers - KeyError: 'eth1'
============================================================ 1 failed, 9 passed, 3 warnings in 0.60s ============================================================
This is is occurring because the code to iterate over the interfaces
when configuring interface-specific DNS is using the original
interface name, not the one from the "set-name" directive. The
following patch corrects the issue:
diff --git a/cloudinit/net/network_state.py b/cloudinit/net/network_state.py
index 95b064f0..06ff8e96 100644
--- a/cloudinit/net/network_state.py
+++ b/cloudinit/net/network_state.py
@@ -543,7 +543,11 @@ class NetworkStateInterpreter(metaclass=CommandHandlerMeta):
def _handle_individual_nameserver(self, command, iface):
_iface = self._network_state.get('interfaces')
nameservers, search = self._parse_dns(command)
- _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
+ try:
+ _iface[iface]['dns'] = {'nameservers': nameservers, 'search': search}
+ except:
+ print("original iface name: %s\niface dict: %s\n" % (iface, _iface))
+ raise
@ensure_command_keys(['destination'])
def handle_route(self, command):
diff --git a/cloudinit/net/tests/test_network_state.py b/cloudinit/net/tests/test_network_state.py
index 84e8308a..45e99171 100644
--- a/cloudinit/net/tests/test_network_state.py
+++ b/cloudinit/net/tests/test_network_state.py
@@ -52,6 +52,7 @@ network:
eth1:
match:
macaddress: '66:77:88:99:00:11'
+ set-name: "ens92"
nameservers:
search: [foo.local, bar.local]
addresses: [4.4.4.4]
Now the above test passes. A PR will be opened on Cloud-Init's GitHub
repository with the above patch.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1946493/+subscriptions
Follow ups
