sts-sponsors team mailing list archive

[Mauricio Oliveira <mauricio.oliveira@xxxxxxxxxxxxx>]

Just to close the loop on this (the d-i build failure due to the
regression in kmod),
this has been reported on LP#1889297, and the fix is already in bionic-proposed.

The debootstrap changes (original work from SF case) too, and verified
(thanks Matthew!)

So, all good!

On Mon, Jul 27, 2020 at 1:01 PM Mauricio Oliveira
<mauricio.oliveira@xxxxxxxxxxxxx> wrote:
>
> Hi Matthew,
>
> Nice tricks you found out.
>
> So, I checked w/ Łukasz today, and he's more aware of the d-i problem;
> and welcomed help to fix it.
>
> He would have to rebuild d-i anyway to bump the kernels, so fixing it
> is on their critical path too.
> (Thus we don't have to worry about having our fix for d-i/kmod/systemd
> being neglected :-)
>
> As for timing for release, now that I have more energy to think about it,
> we don't need the d-i rebuilt to actually pick debootstrap in the ISO
> (which is the timing sensitive part),
> just the new/fixed package available in the archive.
> Because debootstrap does not go into the initrd (which is what d-i
> builds), but in the ISO "archive" only (I guess it's debian-cd which
> does it.)
> [see below]
>
> And similarly, it seems that the network installer pulls debootstrap
> as well -- it's not in initrd either.
> So, fixing debootstrap in the archive is the only thing needed to get
> it on both ISO and netinst. :-)
>
> Today I've done some more testing w/ debootstrap, for the SRU team to
> be a bit happier,
> and uploaded to B/F.  Łucasz has ACKed. :)
>
> I've got to go for the day, but should be able to check MM in a few hours.
> (And later today or tomorrow get into the fun of
>
> [ISO checks for debootstrap]
>
> $ sudo mount -o loop,ro ubuntu-18.04.4-server-amd64.iso mnt
>
> $ find mnt/ -name '*debootstrap*'
> mnt/pool/main/d/debootstrap
> mnt/pool/main/d/debootstrap/debootstrap-udeb_1.0.95ubuntu0.5_all.udeb
>
> $ mkdir initrd
> $ cd initrd
> $ gzip -dc ../mnt/install/initrd.gz | cpio -H newc -mid
> $ find . -name '*debootstrap*'
> $
>
> [netinst checks for debootstrap]
>
> $ wget http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
> $ gzip -dc initrd.gz | cpio -H newc -mid
> $ find . -name '*debootstrap*'
> $
>
> On Sun, Jul 26, 2020 at 11:02 PM Matthew Ruffell
> <matthew.ruffell@xxxxxxxxxxxxx> wrote:
> >
> > Hi Mauricio,
> >
> > Thanks for your awesome analysis!
> >
> > > At this point I think that *if* the case/matter is only about the
> > > network installer,
> > > not the ISO (which is indeed critical with 18.04.5), then it can be
> > > fixed via SRU,
> > > without any concerns.
> >
> > > Would you please evaluate this w/ customer/Lukasƶ on Monday?
> >
> > The customer was not interested in the mini.iso image, they use the network
> > installer, with vmlinuz and initrd.gz. So, that's good to hear that it can be
> > fixed via SRU without any concerns.
> >
> > I suppose there is no need to rush now.
> >
> > > You may notice that even before the --add-udeb flag removal,
> > > libkmod2-udeb *did not*
> > > ship that shared library!
> > >
> > > I wondered how the installer worked. It DOES have that library in
> > > /lib/libkmod.so.2.
> > > And I have no idea where it pulled it from.
> >
> > > I checked MANIFEST files, d-i build log, even download all udebs in
> > > bionic/bionic-updates,
> > > extracted, but nothing shipped it.
> >
> > > This might be some d-i build magic, which I didn't look/find yet.
> >
> > You and me both! I hit the problem when I was making the test mini.iso and
> > test vmlinuz / initrd.gz files. What I did to workaround it was do a grep over
> > the local repository cache d-i keeps when building images, and then do a cheeky
> > sed -i s/libkmod2/libkmod2-udeb/g, commands below:
> >
> > "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27),
> > libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31),
> > libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g"
> > ./apt.udeb/state/lists/nz.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages"
> >
> > "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27),
> > libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31),
> > libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g"
> > ./apt.udeb/state/lists/security.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages"
> >
> > I then re-ran "fakeroot make build_monolithic". Apt seemed to be perfectly happy
> > with the changes, even though libkmod2-udeb was missing from the archive, and
> > produced the test install media correctly.
> >
> > I suppose that hack worked since libkmod2-udeb didn't actually ship
> > any libraries,
> > and /lib/libkmod.so.2 turns up through another magical mechanism.
> >
> > As for that magical mechanism, I found this in the README file in
> > debian-installer/build/README:
> >
> > * Apt is used to download the required udebs. This does *not* include
> >   libraries; libraries used by udebs must be installed on the build system,
> >   and so are build-depended on.
> >
> > Sounds to me that if we include libkmod2, it copies the library from the build
> > host into the install image. That's cheating =p
> >
> > As for the fix, well, I think the path of least resistance is to simply revert
> > the change to libkmod2, and get it to start building libkmod2-udeb
> > again (in bionic).
> >
> > Sure, it will incorrectly advertise itself as containing something useful when
> > it really doesn't, but everything was fine for the two years it existed in the
> > archive. I wouldn't revert it on anything beyond Bionic though.
> >
> > I'll send the above to Lukasz in case he has seen the dependency problem and
> > had been looking into it.
> >
> > Thanks,
> > Matthew
> >
> > On Sat, Jul 25, 2020 at 12:49 PM Mauricio Oliveira
> > <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> > >
> > > Hi Matthew,
> > >
> > > Long story short, the debian-installer rebuild on PPA fails on Bionic.
> > >
> > > The fix for this should apparently involve changes to kmod and systemd,
> > > at least systemd, and I think this may get push back from release team
> > > at this time before the point release (however small the change is.)
> > >
> > > If the fix is needed only in the network-installer, that can be an SRU later.
> > >
> > > Let me explain, and hopefully you can start addressing it with Lukasƶ
> > > on your Monday, before I join.
> > >
> > > The d-i build on PPA fail because 'udev-udeb' cannot pull 'libkmod2'.
> > >
> > >     Building dependency tree...
> > >       Installing libslang2-udeb as Depends of cdebconf-newt-udeb
> > >       udev-udeb:amd64 Depends on libkmod2:amd64 < none @un H > can't
> > > be satisfied!
> > >
> > >     ...
> > >
> > >     The following packages have unmet dependencies:
> > >      udev-udeb : Depends: libkmod2 but it is not installable
> > >     E: Unable to correct problems, you have held broken packages.
> > >     Makefile:620: recipe for target
> > > 'stamps/get_udebs-cdrom_isolinux-stamp' failed
> > >     make[7]: *** [stamps/get_udebs-cdrom_isolinux-stamp] Error 100
> > >
> > > This is correct, because the d-i build does configure sources only for udeb
> > > packages, and libkmod2 is a regular deb.
> > >
> > > Previously it worked because udev-udeb pulled libkmod2-udeb (note suffix.)
> > >
> > > This is a regression in the dependencies of udev-udeb in 237-3ubuntu10.40:
> > >
> > >     https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.39
> > >         Depends on: libkmod2-udeb
> > >
> > >     https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.40
> > >         Depends on: libkmod2
> > >
> > > This happened because the build dependency kmod (which you're familiar with :)
> > > changed versions in between, 24-1ubuntu3.2 to 3.3:
> > >
> > >     On u10.39: Get:45 http://ftpmaster.internal/ubuntu
> > > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.2 [40.0 kB]
> > >     On u10.40: Get:45 http://ftpmaster.internal/ubuntu
> > > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.3 [40.1 kB]
> > >
> > > And this change caused this:
> > >
> > >     kmod (24-1ubuntu3.3) bionic; urgency=medium
> > >     ...
> > >       * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
> > >         actually contain a library. (Closes: #939779)
> > >     ...
> > >
> > > This still builds/ships the libkmod2-udeb package, but...
> > >
> > > The lack of dh_makeshlibs with --add-udeb causes the libkmod2
> > > (non-udeb)'s "shlibs"
> > > file to no longer indicate that udeb packages which depend on that
> > > library package
> > > should depend on the -udeb package instead (exactly what happens here.)
> > >
> > > From dh_makeshlibs(1):
> > >
> > >        --add-udeb=udeb
> > >            Create an additional line for udebs in the shlibs file and
> > > use udeb as the package name for udebs to depend on instead of the
> > > regular library package.
> > >
> > > This can be confirmed with:
> > >
> > >     $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.2
> > >     $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.3
> > >
> > >     $ dpkg-deb --control libkmod2_24-1ubuntu3.2_amd64.deb libkmod2-32-control
> > >     $ dpkg-deb --control libkmod2_24-1ubuntu3.3_amd64.deb libkmod2-33-control
> > >
> > >     $ cat libkmod2-32-control/shlibs
> > >     libkmod 2 libkmod2
> > >     udeb: libkmod 2 libkmod2-udeb
> > >
> > >     $ cat libkmod2-33-control/shlibs
> > >     libkmod 2 libkmod2
> > >
> > > And according to systemd/debian/control file's entry for udev-udeb:
> > >
> > >     Package: udev-udeb
> > >     ...
> > >     Depends: ${shlibs:Depends},
> > >          ${misc:Depends},
> > >          util-linux-udeb
> > >
> > > That dependency comes from ${shlibs:Depends}, handled by dh_shlibdeps(1):
> > >
> > >     https://www.debian.org/doc/manuals/maint-guide/dreq.en.html
> > >     """
> > >     dh_shlibdeps(1) calculates shared library dependencies for binary packages.
> > >     It generates a list of ELF executables and shared libraries it has
> > > found for each binary package.
> > >     This list is used for substituting ${shlibs:Depends}.
> > >     """
> > >
> > > So,
> > >
> > > Since d-i needs both udev-udeb and libkmod2-udeb (for udev,
> > > load/remove modules, etc)
> > > fixing the udev-udeb dependency would require *at least* a
> > > change/rebuild to systemd
> > > to update the Depends: field of udev-udeb.
> > >
> > > Say, even if we came up with a hack/solution that doesn't need a
> > > change/fix to libkmod2.
> > >
> > > This is where I think the SRU/release team might prefer to hold things
> > > for release.
> > > (systemd.)
> > >
> > > At this point I think that *if* the case/matter is only about the
> > > network installer,
> > > not the ISO (which is indeed critical with 18.04.5), then it can be
> > > fixed via SRU,
> > > without any concerns.
> > >
> > > Would you please evaluate this w/ customer/Lukasƶ on Monday?
> > >
> > >
> > > A bit on libkmod2 change:
> > >
> > > The change/regression of --add-udeb to libkmod2 is not incorrect on
> > > its own, since
> > > libkmod2-udeb doesn't ship a shared library.
> > >
> > > I discussed this a bit with Rafael Tinoco (which introduced the
> > > change), and perhaps
> > > the change can be reverted (I'd have to check with a rebuild of kmod
> > > w/ it reverted.)
> > >
> > > (then rebuild kmod, then systemd, then d-i -- after debootstrap w/
> > > your fix, of course.
> > > See it's a chain that looks big for this time before the release. :)
> > >
> > >
> > > Some tech details:
> > >
> > > udev-udeb depends on libkmod2 shared library because of udevadm, for
> > > the kmod built-in.
> > >
> > > You may notice that even before the --add-udeb flag removal,
> > > libkmod2-udeb *did not*
> > > ship that shared library!
> > >
> > > I wondered how the installer worked. It DOES have that library in
> > > /lib/libkmod.so.2.
> > > And I have no idea where it pulled it from.
> > >
> > > I checked MANIFEST files, d-i build log, even download all udebs in
> > > bionic/bionic-updates,
> > > extracted, but nothing shipped it.
> > >
> > > This might be some d-i build magic, which I didn't look/find yet.
> > >
> > >
> > > Finally, I'm a bit tired to think now, so I'll give you the ball for now to play
> > > a bit on considerations of 1) should this really be fixed before
> > > release for ISO,
> > > or just network installer is OK?   and more fun in 2) how to properly
> > > fix this :)
> > >
> > > Thanks and have a great weekend!
> > >
> > > P.S.: I'm attaching my hacky notes about this today, in case it might help.
> > >
> > > On Fri, Jul 24, 2020 at 7:27 PM Mauricio Oliveira
> > > <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> > > >
> > > > Matthew,
> > > >
> > > > By the way, per the SF case, the customer uses the network install method.
> > > > This can be fixed post-release, as SRUs to debian-installer will
> > > > produce new netboot images.
> > > >
> > > > Hopefully they (and other people) don't plan to try to use HTTPS on
> > > > the _ISO_ installer?
> > > > The ISO image is what is more strictly tied to the release date.
> > > > I'm not sure how much that scenario makes sense.
> > > > (there may be the case of trying to use HTTPS for the ISO installer to
> > > > download updates though?)
> > > >
> > > > I'm saying this because this is now (of course!) heading into systemd :-)
> > > > specifically the dependencies of the udev-udeb package.  And that may
> > > > be just an adventure
> > > > to try to fix / SRU before release date. :)
> > > >
> > > > I'll give more details later.
> > > >
> > > > cheers,
> > > >
> > > >
> > > > On Fri, Jul 24, 2020 at 6:30 PM Mauricio Oliveira
> > > > <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Definitely looking at it right now. :)
> > > > >
> > > > > The debootstrap portion is OK. However,
> > > > >
> > > > > there's a PPA build error with the required rebuild of d-i in Bionic
> > > > > (not on Focal -- there's always fun on the releases we actually need!)
> > > > > due to udev-udeb changing dependencies from libkmod2-udeb to libkmod2
> > > > > (non-udeb), which I just tracked down to kmod 24-1ubuntu3.3.
> > > > >
> > > > > I'll provide a write up on the LP bug (and mention internally in SF
> > > > > case) and hopefully it should be relatively simple to fix.
> > > > >
> > > > > As mentioned, debootstrap looks good; so I'll move that one soon.
> > > > >
> > > > > cheers,
> > > > >
> > > > > On Fri, Jul 24, 2020 at 6:25 PM Dan Streetman
> > > > > <dan.streetman@xxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > > Hi Matthew,
> > > > > >
> > > > > > As @mfo has done work in this (installer image) area before (LP:
> > > > > > #1807023) I asked him to take a look; I'm cc'ing our sts sponsors
> > > > > > email list (which includes Mauricio) also.
> > > > > >
> > > > > > I think he's handling the sponsoring, so hopefully it'll make it for
> > > > > > the point release!
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Fri, Jul 24, 2020 at 1:14 AM Matthew Ruffell
> > > > > > <matthew.ruffell@xxxxxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > Hi Dan and Eric,
> > > > > > >
> > > > > > > [URGENT] Please review and sponsor LP1772556. This needs to make 18.04.5 LTS.
> > > > > > >
> > > > > > > https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1772556
> > > > > > >
> > > > > > > I have written to Lukasz Zemczak and made him aware of this bug, and I have also
> > > > > > > asked him to review and sponsor the debdiffs, since he was the last to modify
> > > > > > > debootstrap in Bionic, and is also in charge of producing 18.04.5 LTS images.
> > > > > > >
> > > > > > > I haven't heard back from Lukasz though. I emailed him on my Wednesday, and
> > > > > > > on my Thursday.
> > > > > > >
> > > > > > > Could you please maybe ping Lukasz during your day to make sure he got my email?
> > > > > > >
> > > > > > > The debdiffs in the LP bug are kinda weird. debootstrap uses "3.0 (native)"
> > > > > > > instead of "3.0 (quilt)", and the debdiff seems to take diffs of all the ubuntu
> > > > > > > releases, even though they are all symlinks to a single file that got changed,
> > > > > > > which is "gutsy".
> > > > > > >
> > > > > > > Anyway, please nudge Lukasz, or sponsor my debdiffs. Konica Minolta filed a
> > > > > > > L2 about this in SF289200, and it needs to make 18.04.5 LTS or it will never be
> > > > > > > fixed.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Matthew
> > > > > >
> > > > > > --
> > > > > > Mailing list: https://launchpad.net/~sts-sponsors
> > > > > > Post to     : sts-sponsors@xxxxxxxxxxxxxxxxxxx
> > > > > > Unsubscribe : https://launchpad.net/~sts-sponsors
> > > > > > More help   : https://help.launchpad.net/ListHelp
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Mauricio Faria de Oliveira
> > > >
> > > >
> > > >
> > > > --
> > > > Mauricio Faria de Oliveira
> > >
> > >
> > >
> > > --
> > > Mauricio Faria de Oliveira
>
>
>
> --
> Mauricio Faria de Oliveira



-- 
Mauricio Faria de Oliveira