← Back to team overview

group.of.nepali.translators team mailing list archive

[Bug 1766534] Re: libguestfs cannot start the qemu process on s390x

 

** Changed in: ubuntu-z-systems
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1766534

Title:
  libguestfs cannot start the qemu process on s390x

Status in Ubuntu on IBM z Systems:
  Fix Released
Status in libguestfs package in Ubuntu:
  Fix Released
Status in libguestfs source package in Xenial:
  Won't Fix
Status in libguestfs source package in Artful:
  Won't Fix
Status in libguestfs source package in Bionic:
  Fix Released

Bug description:
  [Impact]

   * the way libguestfs launches libvirt is not working on s390x

   * backport fixes to work on that platform as well

   * The TL;DR of the changes is to add arch specific options to the
  calls being made

  [Test Case]

   * Note: IBM will assist in verifying those and already has on the PPA
   * To run yourself from the original description you can use libguestfs-test-tool:
    1. apt install libguestfs-tools python-libguestfs linux-image-generic qemu
    2. sudo libguestfs-test-tool
    This will by default break on e.g.:
      qemu-system-s390x: -device virtio-rng-pci,rng=rng0: MSI-X support is mandatory in the S390 architecture
      libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages
    3. (upgrade from ppa or proposed depending on what you test)
    4. sudo libguestfs-test-tool
      [...]
      ===== TEST FINISHED OK =====

  [Regression Potential]

   * This modifies how arguments are created, but it does so only for s390x so the regression should be limited to that architecture.
     The issue would be that it doesn't start properly, but since that is the current state there is no real regression - it would just replacve one way of failing for another - and since the tests by IBM on the PPA were good we are convinced that even that regression-risk would not trigger.

   * The FTBFS is a build time only change and should not trigger
  regressions later on.

  [Other Info]

   * Along the fix requested this fixes an FTBFS in Bionic due to the
  move to a new default JDK version.

  ---

  The libguestfs versions < 1.37.22 don't work on s390x, as they don't
  contain the patches which include the s390x specifics to instruct qemu-kvm
  correctly. Ubuntu 16.04 contains an older version than that, which makes
  libguestfs not usable on s390x.

  Steps to reproduce
  ==================
  A chronological list of steps which will bring off the issue I noticed:

      mz@s390xhost$ docker run -it --privileged --rm ubuntu:xenial bash
      root@5dde0aef5d2e:/#
      root@5dde0aef5d2e:/#
      root@5dde0aef5d2e:/# apt update
      root@5dde0aef5d2e:/# DEBIAN_FRONTEND=noninteractive apt install -y \
      libguestfs-tools \
      python-libguestfs \
      linux-image-generic \
      qemu
      root@5dde0aef5d2e:/#
      root@5dde0aef5d2e:/#
      root@5dde0aef5d2e:/# export LIBGUESTFS_DEBUG=1
      root@5dde0aef5d2e:/# export LIBGUESTFS_TRACE=1
      root@5dde0aef5d2e:/# libguestfs-test-tool

  Expected result
  ===============
  The libguestfs-test-tool passes and I can use it on s390x.

  To check if qemu on s390x works independently from libguestfs, I use this
  command:

      root@5dde0aef5d2e:/# qemu-system-s390x \
      -enable-kvm \
      -nographic \
      -kernel /boot/vmlinuz-4.4.0-119-generic \
      -initrd /boot/initrd.img-4.4.0-119-generic \
      -m 1G \
      -M s390-ccw-virtio

      [...]  # boot

      BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
      Enter 'help' for a list of built-in commands.

      (initramfs)

  Actual result
  =============

  libguestfs instructs qemu to use pci devices and serial consoles
  which doesn't work on s390x, e.g.:

      root@5dde0aef5d2e:/# libguestfs-test-tool
           ************************************************************
           *                    IMPORTANT NOTICE
           *
           * When reporting bugs, include the COMPLETE, UNEDITED
           * output below in your bug report.
           *
           ************************************************************
      libguestfs: trace: set_verbose true
      libguestfs: trace: set_verbose = 0
      libguestfs: trace: set_verbose true
      libguestfs: trace: set_verbose = 0
      LIBGUESTFS_DEBUG=1
      LIBGUESTFS_TRACE=1
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      SELinux: sh: 1: getenforce: not found
      libguestfs: trace: add_drive_scratch 104857600
      libguestfs: trace: get_tmpdir
      libguestfs: trace: get_tmpdir = "/tmp"
      libguestfs: trace: disk_create "/tmp/libguestfs82Gj1d/scratch.1" "raw" 104857600
      libguestfs: trace: disk_create = 0
      libguestfs: trace: add_drive "/tmp/libguestfs82Gj1d/scratch.1" "format:raw" "cachemode:unsafe"
      libguestfs: trace: add_drive = 0
      libguestfs: trace: add_drive_scratch = 0
      libguestfs: trace: get_append
      libguestfs: trace: get_append = "NULL"
      guestfs_get_append: (null)
      libguestfs: trace: get_autosync
      libguestfs: trace: get_autosync = 1
      guestfs_get_autosync: 1
      libguestfs: trace: get_backend
      libguestfs: trace: get_backend = "direct"
      guestfs_get_backend: direct
      libguestfs: trace: get_backend_settings
      libguestfs: trace: get_backend_settings = []
      guestfs_get_backend_settings: []
      libguestfs: trace: get_cachedir
      libguestfs: trace: get_cachedir = "/var/tmp"
      guestfs_get_cachedir: /var/tmp
      libguestfs: trace: get_direct
      libguestfs: trace: get_direct = 0
      guestfs_get_direct: 0
      libguestfs: trace: get_hv
      libguestfs: trace: get_hv = "/usr/bin/qemu-system-s390x"
      guestfs_get_hv: /usr/bin/qemu-system-s390x
      libguestfs: trace: get_memsize
      libguestfs: trace: get_memsize = 500
      guestfs_get_memsize: 500
      libguestfs: trace: get_network
      libguestfs: trace: get_network = 0
      guestfs_get_network: 0
      libguestfs: trace: get_path
      libguestfs: trace: get_path = "/usr/lib/s390x-linux-gnu/guestfs"
      guestfs_get_path: /usr/lib/s390x-linux-gnu/guestfs
      libguestfs: trace: get_pgroup
      libguestfs: trace: get_pgroup = 0
      guestfs_get_pgroup: 0
      libguestfs: trace: get_program
      libguestfs: trace: get_program = "libguestfs-test-tool"
      guestfs_get_program: libguestfs-test-tool
      libguestfs: trace: get_recovery_proc
      libguestfs: trace: get_recovery_proc = 1
      guestfs_get_recovery_proc: 1
      libguestfs: trace: get_selinux
      libguestfs: trace: get_selinux = 0
      guestfs_get_selinux: 0
      libguestfs: trace: get_smp
      libguestfs: trace: get_smp = 1
      guestfs_get_smp: 1
      libguestfs: trace: get_tmpdir
      libguestfs: trace: get_tmpdir = "/tmp"
      guestfs_get_tmpdir: /tmp
      libguestfs: trace: get_trace
      libguestfs: trace: get_trace = 1
      guestfs_get_trace: 1
      libguestfs: trace: get_verbose
      libguestfs: trace: get_verbose = 1
      guestfs_get_verbose: 1
      host_cpu: s390x
      Launching appliance, timeout set to 600 seconds.
      libguestfs: trace: launch
      libguestfs: trace: version
      libguestfs: trace: version = <struct guestfs_version *>
      libguestfs: trace: get_backend
      libguestfs: trace: get_backend = "direct"
      libguestfs: launch: program=libguestfs-test-tool
      libguestfs: launch: version=1.32.2
      libguestfs: launch: backend registered: unix
      libguestfs: launch: backend registered: uml
      libguestfs: launch: backend registered: libvirt
      libguestfs: launch: backend registered: direct
      libguestfs: launch: backend=direct
      libguestfs: launch: tmpdir=/tmp/libguestfs82Gj1d
      libguestfs: launch: umask=0022
      libguestfs: launch: euid=0
      libguestfs: trace: get_backend_setting "force_tcg"
      libguestfs: trace: get_backend_setting = NULL (error)
      libguestfs: trace: get_cachedir
      libguestfs: trace: get_cachedir = "/var/tmp"
      libguestfs: [00000ms] begin building supermin appliance
      libguestfs: [00000ms] run supermin
      libguestfs: command: run: /usr/bin/supermin
      libguestfs: command: run: \ --build
      libguestfs: command: run: \ --verbose
      libguestfs: command: run: \ --if-newer
      libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
      libguestfs: command: run: \ --copy-kernel
      libguestfs: command: run: \ -f ext2
      libguestfs: command: run: \ --host-cpu s390x
      libguestfs: command: run: \ /usr/lib/s390x-linux-gnu/guestfs/supermin.d
      libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
      supermin: version: 5.1.14
      supermin: package handler: debian/dpkg
      supermin: acquiring lock on /var/tmp/.guestfs-0/lock
      supermin: build: /usr/lib/s390x-linux-gnu/guestfs/supermin.d
      supermin: reading the supermin appliance
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages type uncompressed packages
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-hfsplus type uncompressed packages
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-reiserfs type uncompressed packages
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/packages-xfs type uncompressed packages
      supermin: build: visiting /usr/lib/s390x-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
      supermin: mapping package names to installed packages
      supermin: resolving full list of package dependencies
      supermin: build: 194 packages, including dependencies
      supermin: build: 6751 files
      supermin: build: 3694 files, after matching excludefiles
      supermin: build: 3696 files, after adding hostfiles
      supermin: build: 3696 files, after removing unreadable files
      supermin: build: 3700 files, after munging
      supermin: kernel: picked kernel vmlinuz-4.4.0-119-generic
      supermin: kernel: picked modules path /lib/modules/4.4.0-119-generic
      supermin: kernel: kernel_version 4.4.0-119-generic
      supermin: kernel: modules /lib/modules/4.4.0-119-generic
      supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.s5hkw6ja/root'
      supermin: ext2: populating from base image
      supermin: ext2: copying files from host filesystem
      supermin: ext2: copying kernel modules
      supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.s5hkw6ja/initrd'
      supermin: ext2: wrote 9 modules to minimal initrd
      supermin: renaming /var/tmp/.guestfs-0/appliance.d.s5hkw6ja to /var/tmp/.guestfs-0/appliance.d
      libguestfs: [01090ms] finished building supermin appliance
      libguestfs: [01090ms] begin testing qemu features
      libguestfs: command: run: /usr/bin/qemu-system-s390x
      libguestfs: command: run: \ -display none
      libguestfs: command: run: \ -help
      libguestfs: command: run: /usr/bin/qemu-system-s390x
      libguestfs: command: run: \ -display none
      libguestfs: command: run: \ -version
      libguestfs: qemu version 2.5
      libguestfs: command: run: /usr/bin/qemu-system-s390x
      libguestfs: command: run: \ -display none
      libguestfs: command: run: \ -machine accel=kvm:tcg
      libguestfs: command: run: \ -device ?
      libguestfs: [01166ms] finished testing qemu features
      libguestfs: trace: get_backend_setting "gdb"
      libguestfs: trace: get_backend_setting = NULL (error)
      [01172ms] /usr/bin/qemu-system-s390x \
          -global virtio-blk-pci.scsi=off \
          -nodefconfig \
          -enable-fips \
          -nodefaults \
          -display none \
          -machine accel=kvm:tcg \
          -cpu host \
          -m 500 \
          -no-reboot \
          -rtc driftfix=slew \
          -global kvm-pit.lost_tick_policy=discard \
          -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
          -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
          -device virtio-scsi-pci,id=scsi \
          -drive file=/tmp/libguestfs82Gj1d/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \
          -device scsi-hd,drive=hd0 \
          -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \
          -device scsi-hd,drive=appliance \
          -device virtio-serial-pci \
          -serial stdio \
          -chardev socket,path=/tmp/libguestfs82Gj1d/guestfsd.sock,id=channel0 \
          -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
          -append 'panic=1 console=ttyS0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm'
      qemu-system-s390x: Warning: global kvm-pit.lost_tick_policy has invalid class name
      Alarm clock

  Environment
  ===========

  This affects ubuntu 16.04 (see below)

  host
  ----

      $ lsb_release -a
      LSB Version:	core-9.20160110ubuntu0.2-noarch:core-9.20160110ubuntu0.2-s390x:printing-9.20160110ubuntu0.2-noarch:printing-9.20160110ubuntu0.2-s390x:security-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-s390x
      Distributor ID:	Ubuntu
      Description:	Ubuntu 16.04.4 LTS
      Release:	16.04
      Codename:	xenial

  container
  ---------

      root@5dde0aef5d2e:/# lsb_release -a
      No LSB modules are available.
      Distributor ID:	Ubuntu
      Description:	Ubuntu 16.04.3 LTS
      Release:	16.04
      Codename:	xenial

      root@5dde0aef5d2e:/# apt-cache policy libguestfs-tools
      libguestfs-tools:
        Installed: 1:1.32.2-4ubuntu2
        Candidate: 1:1.32.2-4ubuntu2
        Version table:
       *** 1:1.32.2-4ubuntu2 500
              500 http://ports.ubuntu.com/ubuntu-ports xenial/universe s390x Packages
              100 /var/lib/dpkg/status

  Additional information
  ======================

  Please be aware that there this is a known libguestfs upstream bug [1]
  which is already fixed and backported by RedHat [2][3] and was the
  trigger for a patch series on the libguestfs mailing list [4]. As an
  example, the latest known needed patch is [5]. As a smoke test, in case
  the libguestfs-test-tool result was a false positive, I used the python
  binding code in Appendix A below, which is based on [6].

  FWIW, I checked Ubuntu 18.04 (beta), and it doesn't work there too.
  The packaged version of libguestfs is v1.36 in 18.04 bionic beaver,
  which is too old (and/or doesn't contain the needed backports).

  References
  ==========
  [1] Bug 1376547 - qemu-system-s390x: -device isa-serial,chardev=charserial0,
      id=serial0: 'isa-serial' is not a valid device model name
      https://bugzilla.redhat.com/show_bug.cgi?id=1376547
  [2] [Private] Bug 1479526 - [RFE] libguestfs for z Systems
      https://bugzilla.redhat.com/show_bug.cgi?id=1479526
  [3] The RHEL7.5 branch of libguestfs with backports to v1.36
      https://github.com/libguestfs/libguestfs/tree/rhel-7.5
  [4] The patch series of libguestfs on s390x:
      https://www.redhat.com/archives/libguestfs/2017-May/msg00066.html
  [5] Fix of an x86 specific item to enable s390x in v1.37.22
      https://github.com/libguestfs/libguestfs/commit/5b60dd4eff02f48d344bcdad0d4bad4676ca9168
  [6] https://rwmj.wordpress.com/2010/02/01/libguestfs-python-example/

  Appendix A
  ==========

  Smoke test, based on the example from the author of libguestfs [6]:

      root@26248e573a87:/# cat smoke.py
      #!/usr/bin/python

      # To see the Python docs, do: help (guestfs)
      # To see the general docs http://libguestfs.org/guestfs.3.html
      # For FAQ see http://libguestfs.org/FAQ.html
      import os
      import guestfs
      g = guestfs.GuestFS()
      f = open("/tmp/test.img", "w")
      f.truncate(500 * 1024 * 1024)
      f.close()
      g.add_drive("/tmp/test.img")
      g.launch()
      g.part_disk("/dev/sda", "mbr")
      g.mkfs("ext2", "/dev/sda1")
      g.mount("/dev/sda1", "/")
      f = open("/tmp/ks.cfg", "w")
      f.write("# this would be the kickstart file\n")
      f.write("#\n")
      f.write ("# end of file\n")
      f.close()
      g.upload("/tmp/ks.cfg", "/ks.cfg")
      g.sync()
      g.umount_all()

  == Comment: #1 - Markus Zoeller <mzoeller@xxxxxxxxxx> - 2018-04-20 06:51:10 ==
  With the Debian Sid repository I could install libguestfs v1.38 in an
  Ubuntu 18.04 (bionic beaver) container, all on s390x:

      [mz@s390xhost]$ docker run -it --privileged --rm s390x/ubuntu:18.04 bash
      $ apt update
      $ apt install -y vim
      $ apt-cache policy libguestfs-tools
      $ apt install -y software-properties-common \
      debian-archive-keyring \
      dirmngr
      $ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553
      $ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010
      $ add-apt-repository "deb http://ftp.de.debian.org/debian sid main"

      $ cat << EOF > /etc/apt/preferences.d/libguestfs.pref
      # Note: 2 blank lines are required between entries
      Package: *
      Pin: release a=bionic
      Pin-Priority: 500

      Package: *
      Pin: origin "ftp.de.debian.org"
      Pin-Priority: 300

      # Pattern includes 'python-guestfs' and 'libguestfs-tools' and similarily named dependencies
      Package: *guestfs*
      Pin: origin "ftp.de.debian.org"
      Pin-Priority: 700
      EOF
      $ cat /etc/apt/preferences.d/libguestfs.pref

      $ apt update
      $ apt-cache policy libguestfs-tools

      $ DEBIAN_FRONTEND=noninteractive apt install -y \
          libguestfs-tools \
          python-libguestfs \
          linux-image-generic \
          qemu-system \
          qemu-kvm

      $ apt-cache policy libguestfs-tools
      $ export LIBGUESTFS_DEBUG=1
      $ export LIBGUESTFS_TRACE=1
      $ libguestfs-test-tool
      [...]
      ===== TEST FINISHED OK =====

  This verifies that libguestfs v1.38 has the needed code to work on
  s390x.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1766534/+subscriptions