ubuntu-public-cloud team mailing list archive

[Miriam España Acebal <2116077@xxxxxxxxxxxxxxxxxx>]

** Changed in: azure-vm-utils (Ubuntu)
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Public Cloud, which is subscribed to azure-vm-utils in Ubuntu.
https://bugs.launchpad.net/bugs/2116077

Title:
  [SRU] azure-vm-utils updates

Status in azure-vm-utils package in Ubuntu:
  Fix Released
Status in azure-vm-utils source package in Jammy:
  Fix Released
Status in azure-vm-utils source package in Noble:
  Fix Released
Status in azure-vm-utils source package in Plucky:
  Fix Released

Bug description:
  [ Impact ]

    This release contains both bug-fixes and new
    features and we would like to make sure all of our supported customers have
    access to these improvements.

    Full release notes are available at:
      https://github.com/Azure/azure-vm-utils/releases/tag/v0.6.0

  [ Test Plan ]

    A) Ensure the selftests run at autopkgtest time, and pass.
    B) Check installation/upgrade/removal in Azure VM machine
    C) Manual testing in Azure VM machines (hardware dependent) composed of
       1. For any MANA, mlx4, or mlx5 SR-IOV devices with the IFF_SLAVE flag
          set:
          (a) The appropriate udev properties should be configured; and
          (b) systemd-networkd should report these devices as unmanaged

       2. Any network devices not matching the above criteria are left
          unaffected.
       3. Manually check that the command `DEVNAME=<nvme device name> azure-
          nvme-id --udev gives correct output for some device. Only Direct
          Disk v2 is fully supported for now.
       4. Check that expected /dev/disk/azure symlinks are created for the
          device.
       5. Use unmkintramfs to ensure that the udev rules are copied to the
          initrd correctly.

    Detailed commands can be found below. A) and B) are not detailed - it is
    understood that the reviewer or tester knows how to do it.

    Results of this test plan should be attached to the bug in comments per
    target series of the SRU for the SRU Team (Andreas Hasenack) to review.

    Requeriments:

    Some azure VM are required to test this package, as it contains
          specific configurations for Ubuntu on Azure. An Azure account and
          access to the Azure portal is needed.

    azure-cli package is needed for command line VM creation (
  https://packages.microsoft.com/repos/azure-cli/ ).

    Preparing the manual testing:

    ## 0.0 ## AZURE VM CREATION: selection of VM's family size depending
  on what disk and net driver we need to check:

   For Microsoft NVMe Direct Disk v2, MSFT NVMe Accelerator v1.0 and networking
   on mellanox v5 ->
       az vm create --resource-group miriam-azure-vm-utils --name nmve_direct --image "Canonical:ubuntu-25_10-daily:server:latest" --ssh-key-values ~/.ssh/id_rsa.pub --size Standard_E2ads_v6 --admin-username ubuntu

   For Microsoft NVMe Direct Disk ->
       az vm create --resource-group miriam-azure-vm-utils --name nmve_direct_noversion --image "Canonical:ubuntu-25_10-daily:server:latest" --ssh-key-values ~/.ssh/id_rsa.pub --size Standard_D2alds_v6 --admin-username ubuntu

   For Net mana driver ->
       az vm create --resource-group miriam-azure-vm-utils --name nmvemana --image "Canonical:ubuntu-25_10-daily:server:latest" --ssh-key-values ~/.ssh/id_rsa.pub --size Standard_D2ls_v6 --admin-username ubuntu

    Note: Earlier v2/v3/v4 sizes with AN enabled is most likely to
  result in mlx4, but there's no guarantee. Therefore, we may never have
  the opportunity to test mlx4 for sure.

  Please, log in every machine with ssh ubuntu@<vm_ip> to perform the
  next steps.

         ## 0.1 ## CHECK DEVICES IN ORIGINAL STATE

         # DISKS

         $ nvme list | grep -e "Microsoft NVMe Direct Disk v2" -e "MSFT
  NVMe Accelerator v1.0" -e "Microsoft NVMe Direct Disk"

         # NETWORK

         To record differences before and after installing the package plus
         rebooting, the following command

         $ networkctl status -a -l | grep -v Jul | tee
  net_{before,after}.txt

         was used. Please, change the month in the second part of the command to
         fit the actual month the manual testing is being done. Note that actual
         differences are occasioned by the rebooting of systemd-networkd , not by
         the deployment of the azure-vm-utils package. However, we include the
         step for a sanity check.

         To check driver presence (mana, mlx4, mlx5)

         $ networkctl status $(ip a | grep SLAVE | cut -d':' -f2 |
  xargs) | grep -i driver | grep -e mana -e mlx

         To check udev rules (at this stage, only showing the driver):

         $ udevadm info /sys/class/net/$(ip a | grep SLAVE | cut -d':'
  -f2 | xargs) | grep -e AZURE_UNMANAGED_SRIOV -e ID_NET_MANAGED_BY -e
  ID_NET_DRIVER

         ## 0.2 ## INSTALLING PACKAGE

         $ sudo apt install azure-vm-utils

         checking all went well:

         $ dpkg -l azure-vm-utils | grep ii

         ## 0. 3 ## ENABLING SYSTEMD-NETWORKD IN DEBUG MODE

         $ sudo mkdir -p /etc/systemd/system/systemd-networkd.service.d/

         $ sudo cat > /etc/systemd/system/systemd-networkd.service.d/10-debug.conf <<EOF
  [Service]
  Environment=SYSTEMD_LOG_LEVEL=debug
  EOF

         ## 0. 4 ## REBOOTING

         $ sudo shutdown -r now

    Performing manual testing C):

        ## 1.a ## UDEV CONFIGURED

        $ udevadm info /sys/class/net/$(ip a | grep SLAVE | cut -d':'
  -f2 | xargs) | grep -e AZURE_UNMANAGED_SRIOV -e ID_NET_MANAGED_BY -e
  ID_NET_DRIVER

        ## 1.b ## NETWORKD-SYSTEMD PROCESS THE FILE FOR SRI-OV DEVICES
  AND RETURNS THEM UNMANAGED

        $ sudo journalctl -b -u systemd-networkd | grep azure

        $ sudo journalctl -b -u systemd-networkd | grep $(ip a | grep
  SLAVE | cut -d':' -f2 | xargs) | grep -e SLAVE -e unmanaged

        ## 2 ## NO OTHER NETWORKING ITEMS AFFECTED

        $ networkctl status -a -l | grep -v Jul | tee net_after.txt

        $ diff net_before.txt net_after.txt

        The output should be something similar to:

          17c17
    <                    Link File: /usr/lib/systemd/network/99-default.link
     ---
     >                    Link File: /run/systemd/network/10-netplan-eth0.link
     24d23
     <            Alternative Names: enx002248814532
     52a52
     >                               enx002248814532

        ## 3 ## azure-nvme UTIL WORKS FOR NMVE DISK V2

        $ sudo DEVNAME=$(nvme list | grep v2 | cut -d' ' -f1) azure-
  nvme-id --udev

        ## 4 ## SYMLINKS ARE CREATED FOR THE NVME DEVICES

        $ udevadm info $(nvme list | grep "MSFT NVMe Accelerator v1.0" |
  cut -d' ' -f1) | grep -i -e model -e azure

        $ udevadm info $(nvme list | grep "Microsoft NVMe Direct Disk" |
  cut -d' ' -f1) | grep -i -e model -e azure

        $ udevadm info $(nvme list | grep "Microsoft NVMe Direct Disk
  v2" | cut -d' ' -f1) | grep -i -e model -e azure

        ## 5 ## UDEV RULES ARE COPIED TO INITRD

        $ unmkinitramfs /boot/initrd.img-$(uname -r) initramfs/

        $ ls initramfs/lib/udev/rules.d/*0*azure* | wc  -l  # it should
  return only 2

  [ Where problems could occur ]

   Although azure-vm-utils is a package to be used intentionally for Ubuntu on
   Azure, this package could be installed on all Ubuntu systems by users, which
   should remain unaffected by any changes made to this package.

   Sometimes this extends beyond what we might consider supportable
   configurations; we try to stretch ourselves so that no user who apparently
   has an otherwise functional system is affected by this package or changes
   to it.

   This mainly includes regressions caused by the introduction of new udev rules
   that conflict with the existing mapping, specially netplan configurations
   generated by cloud-init.

   The test plan above is intended to foresee this situation and prevent any
   risk. In addition, due to hardware constraints and awareness of the above,
   upstream performs a test with a custom netplan configuration that enables DHCP
   on all Ethernet devices and validates that the interface is unmanaged.

  [ Other Info ]

   The QA process consists of three stages:  pre-SRU tests (the [Test Plan]
   section), SRU tests and SRU Verification.

   Output reports per target series of the [Test Plan] should be attached
   to this bug before them can be reviewed by the SRU team.

   SRU Test Cases
   ..............

   The following will be executed for representative combinations of supported
   architectures, image types and machine sizes:

      1.) Build new cloud image with -proposed package
      2.) Boot machine from image
      3.) Run all CPC image tests against machine

   SRU Verification
   ................

   When a new version of azure-vm-utils is uploaded to -proposed, there will be
   validate actions performed by the CPC azure squad and others from Microsoft
   maintainers. Therefore, the following will be done:

   - By The CPC Azure squad team

     -  the CPC azure squad team will write new automated tests to cover new
        testable functionality (if any) in the new package
     -  the automated testing that the CPC team normally runs against Azure
        images before they are published will be run against the -proposed
        package.
     -  The new package candidate version is built in devel-proposed and tested
        on the target suite. This will involve one or both of:
        -  Installing the devel-proposed packages on an Azure VM, manually
           restoring the VM to a first boot state and rebooting it,
        -  Generating a fresh image with the devel-proposed package version
           preinstalled and testing that directly
     -  Once the manual packaging tests pass successfully and the package
        requires no further changes, it will be marked as such on the tracking
        bug. On the development release, this is done by removing the
        *block-proposed* tag.

   - By the Microsoft team maintaining the Azure VM Utils
  project(upstream QA)

      -  that the new package addresses the issues it is expected to address, and
      -  that the new package passes their internal image validation

   **If appropriate due to the nature of the changes (functional embargo
  on publication), the steps above may be done in a private PPA prior to
  landing in devel-proposed.**

  The CPC team will be responsible for attaching a summary of testing to the bug.
  CPC team members will not mark ‘verification-done’ until this has happened.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/azure-vm-utils/+bug/2116077/+subscriptions