yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1965660] Re: IPoIB interface is not coming up during boot with NetworkManager using ifcfg/sysconfig files

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: James Falcon <1965660@xxxxxxxxxxxxxxxxxx>
Date: Mon, 28 Mar 2022 15:46:49 -0000
Reply-to: Bug 1965660 <1965660@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
I'm going to close this because disabling the network configuration
solves your issue. If that is incorrect, please set status back to New.
The most recent release of cloud-init has proper NetworkManager support.

** Changed in: cloud-init
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1965660

Title:
  IPoIB interface is not coming up during boot with NetworkManager using
  ifcfg/sysconfig files

Status in cloud-init:
  Invalid

Bug description:
  I am trying to create in OpenStack setup a Centos8-Stream VM with IPoIB interfaces.
  image was created with cloud-init and dhcp-all-interfaces.
  And I see that the interface is not configured automatically to be a "connected" interface by NetworkManager during boot and therefore not pulling DHCP.
  Only when I run the command "mcli conn add type infiniband con-name ib0 ifname ib0" the interface becomes active and assigned with DHCP IP.
  When I use Centos7-based image (where network manager was not the default networking service) I dont have this issue.

  cloud-init collect-logs output is attached.

  Some more interesting outputs:

  # cat /etc/os-release
  NAME="CentOS Stream"
  VERSION="8"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="8"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="CentOS Stream 8"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:8"
  HOME_URL="https://centos.org/";
  BUG_REPORT_URL="https://bugzilla.redhat.com/";
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
  REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

  # ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP mode DEFAULT group default qlen 256
      link/infiniband 00:00:01:4a:fe:80:00:00:00:00:00:00:fa:16:3e:00:00:46:de:18 brd 00:ff:ff:ff:ff:12:40:1b:80:65:00:00:00:00:00:00:ff:ff:ff:ff

  # ip addr show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256
      link/infiniband 00:00:01:4a:fe:80:00:00:00:00:00:00:fa:16:3e:00:00:46:de:18 brd 00:ff:ff:ff:ff:12:40:1b:80:65:00:00:00:00:00:00:ff:ff:ff:ff

  # cat /etc/sysconfig/network-scripts/ifcfg-ib0

  # Created by cloud-init on instance boot automatically, do not edit.
  #
  BOOTPROTO=dhcp
  DEVICE=ib0
  HWADDR=00:00:01:4a:fe:80:00:00:00:00:00:00:fa:16:3e:00:00:46:de:18
  ONBOOT=yes
  TYPE=Ethernet
  USERCTL=no

  # ncmli dev show
  bash: ncmli: command not found
  [root@localhost stack]# nmcli dev show
  GENERAL.DEVICE:                         ib0
  GENERAL.TYPE:                           infiniband
  GENERAL.HWADDR:                         00:00:01:4A:FE:80:00:00:00:00:00:00:FA:>
  GENERAL.MTU:                            2044
  GENERAL.STATE:                          30 (disconnected)
  GENERAL.CONNECTION:                     --
  GENERAL.CON-PATH:                       --
  IP4.GATEWAY:                            --
  IP6.GATEWAY:                            --

  GENERAL.DEVICE:                         lo
  GENERAL.TYPE:                           loopback
  GENERAL.HWADDR:                         00:00:00:00:00:00
  GENERAL.MTU:                            65536
  GENERAL.STATE:                          10 (unmanaged)
  GENERAL.CONNECTION:                     --
  GENERAL.CON-PATH:                       --
  IP4.ADDRESS[1]:                         127.0.0.1/8
  IP4.GATEWAY:                            --
  IP6.ADDRESS[1]:                         ::1/128
  IP6.GATEWAY:                            --
  IP6.ROUTE[1]:                           dst = ::1/128, nh = ::, mt = 256

  # nmcli conn show

  #

  # cat /var/log/messages | grep -i networkman
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.1869] NetworkManager (version 1.36.0-0.3.el8) is starting... (for the first time)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.1872] Read config: /etc/NetworkManager/NetworkManager.conf (etc: 00-main.conf)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <warn>  [1647763876.1872] config: unknown key 'autoconnect-retries' in section [connection] of file '/etc/NetworkManager/conf.d/00-main.conf'
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.1887] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.1973] manager[0x561ccff17000]: monitoring kernel firmware directory '/lib/firmware'.
  Mar 20 08:11:16 localhost dbus-daemon[1106]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.7' (uid=0 pid=1239 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2811] hostname: hostname: using hostnamed
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2811] hostname: hostname changed from (none) to "localhost.localdomain"
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2814] dns-mgr[0x561ccfefa250]: init: dns=default,systemd-resolved rc-manager=symlink
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2851] Loaded device plugin: NMTeamFactory (/usr/lib64/NetworkManager/1.36.0-0.3.el8/libnm-device-plugin-team.so)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2851] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2852] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2852] manager: Networking is enabled by state file
  Mar 20 08:11:16 localhost dbus-daemon[1106]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.7' (uid=0 pid=1239 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2863] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.36.0-0.3.el8/libnm-settings-plugin-ifcfg-rh.so")
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2863] settings: Loaded settings plugin: keyfile (internal)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <warn>  [1647763876.2888] ifcfg-rh: load[/etc/sysconfig/network-scripts/ifcfg-ib0]: failure to read file: 802-3-ethernet.mac-address: '00:00:01:4a:fe:80:00:00:00:00:00:00:fa:16:3e:00:00:46:de:18' is not a valid MAC address
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2939] dhcp-init: Using DHCP client 'dhclient'
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2939] device (lo): carrier: link connected
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2941] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2948] manager: (ib0): new InfiniBand device (/org/freedesktop/NetworkManager/Devices/2)
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.2950] device (ib0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.3759] device (ib0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.3796] device (ib0): carrier: link connected
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.3797] manager: startup complete
  Mar 20 08:11:16 localhost NetworkManager[1239]: <info>  [1647763876.8378] hostname: hostname changed from "localhost.localdomain" to "localhost"
  Mar 20 08:11:18 localhost NetworkManager[1239]: <info>  [1647763878.3225] audit: op="reload" arg="0" pid=2166 uid=0 result="success"
  Mar 20 08:11:18 localhost NetworkManager[1239]: <warn>  [1647763878.3229] config: unknown key 'autoconnect-retries' in section [connection] of file '/etc/NetworkManager/conf.d/00-main.conf'
  Mar 20 08:11:18 localhost NetworkManager[1239]: <info>  [1647763878.3229] config: signal: SIGHUP (no changes from disk)
  Mar 20 08:11:18 localhost dbus-daemon[1106]: [system] Activating via systemd: service name='org.freedesktop.resolve1' unit='dbus-org.freedesktop.resolve1.service' requested by ':1.7' (uid=0 pid=1239 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
  Mar 20 08:11:18 localhost NetworkManager[1239]: <info>  [1647763878.3248] policy: set-hostname: set hostname to 'localhost.localdomain' (no hostname found)
  Mar 20 08:11:28 localhost systemd[1]: NetworkManager-dispatcher.service: Succeeded.

  +++++++++++++++++++++++++++++

  # nmcli conn add type infiniband  con-name ib0 ifname ib0
  [  703.180806] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
  [  703.182712] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
  Connection 'ib0' (be895007-5a28-446b-a866-09c02d953ae4) successfully added.
  [root@localhost stack]# [  703.206044] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready

  [root@localhost stack]# nmcli conn show
  NAME  UUID                                  TYPE        DEVICE
  ib0   be895007-5a28-446b-a866-09c02d953ae4  infiniband  ib0
  [root@localhost stack]#
  [root@localhost stack]# ip addr show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 256
      link/infiniband 00:00:01:4a:fe:80:00:00:00:00:00:00:fa:16:3e:00:00:46:de:18 brd 00:ff:ff:ff:ff:12:40:1b:80:65:00:00:00:00:00:00:ff:ff:ff:ff
      inet 11.11.11.47/24 brd 11.11.11.255 scope global dynamic noprefixroute ib0
         valid_lft 86385sec preferred_lft 86385sec
      inet6 fe80::4dac:e4bb:72bf:55b7/64 scope link noprefixroute
         valid_lft forever preferred_lft forever

  
  [root@localhost stack]# cat /etc/sysconfig/network-scripts/ifcfg-ib0-1 
  CONNECTED_MODE=no
  TYPE=InfiniBand
  PROXY_METHOD=none
  BROWSER_ONLY=no
  BOOTPROTO=dhcp
  DEFROUTE=yes
  IPV4_FAILURE_FATAL=no
  IPV6INIT=yes
  IPV6_AUTOCONF=yes
  IPV6_DEFROUTE=yes
  IPV6_FAILURE_FATAL=no
  IPV6_ADDR_GEN_MODE=stable-privacy
  NAME=ib0
  UUID=be895007-5a28-446b-a866-09c02d953ae4
  DEVICE=ib0
  ONBOOT=yes

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1965660/+subscriptions
References

[Bug 1965660] [NEW] IPoIB interface is not coming up during boot (cloud-init/NetworkManager)
From: Itai Levy, 2022-03-20