sts-sponsors team mailing list archive

[Mauricio Oliveira <mauricio.oliveira@xxxxxxxxxxxxx>]

Hi Matthew,

Nice tricks you found out.

So, I checked w/ Łukasz today, and he's more aware of the d-i problem;
and welcomed help to fix it.

He would have to rebuild d-i anyway to bump the kernels, so fixing it
is on their critical path too.
(Thus we don't have to worry about having our fix for d-i/kmod/systemd
being neglected :-)

As for timing for release, now that I have more energy to think about it,
we don't need the d-i rebuilt to actually pick debootstrap in the ISO
(which is the timing sensitive part),
just the new/fixed package available in the archive.
Because debootstrap does not go into the initrd (which is what d-i
builds), but in the ISO "archive" only (I guess it's debian-cd which
does it.)
[see below]

And similarly, it seems that the network installer pulls debootstrap
as well -- it's not in initrd either.
So, fixing debootstrap in the archive is the only thing needed to get
it on both ISO and netinst. :-)

Today I've done some more testing w/ debootstrap, for the SRU team to
be a bit happier,
and uploaded to B/F.  Łucasz has ACKed. :)

I've got to go for the day, but should be able to check MM in a few hours.
(And later today or tomorrow get into the fun of

[ISO checks for debootstrap]

$ sudo mount -o loop,ro ubuntu-18.04.4-server-amd64.iso mnt

$ find mnt/ -name '*debootstrap*'
mnt/pool/main/d/debootstrap
mnt/pool/main/d/debootstrap/debootstrap-udeb_1.0.95ubuntu0.5_all.udeb

$ mkdir initrd
$ cd initrd
$ gzip -dc ../mnt/install/initrd.gz | cpio -H newc -mid
$ find . -name '*debootstrap*'
$

[netinst checks for debootstrap]

$ wget http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/initrd.gz
$ gzip -dc initrd.gz | cpio -H newc -mid
$ find . -name '*debootstrap*'
$

On Sun, Jul 26, 2020 at 11:02 PM Matthew Ruffell
<matthew.ruffell@xxxxxxxxxxxxx> wrote:
>
> Hi Mauricio,
>
> Thanks for your awesome analysis!
>
> > At this point I think that *if* the case/matter is only about the
> > network installer,
> > not the ISO (which is indeed critical with 18.04.5), then it can be
> > fixed via SRU,
> > without any concerns.
>
> > Would you please evaluate this w/ customer/Lukasƶ on Monday?
>
> The customer was not interested in the mini.iso image, they use the network
> installer, with vmlinuz and initrd.gz. So, that's good to hear that it can be
> fixed via SRU without any concerns.
>
> I suppose there is no need to rush now.
>
> > You may notice that even before the --add-udeb flag removal,
> > libkmod2-udeb *did not*
> > ship that shared library!
> >
> > I wondered how the installer worked. It DOES have that library in
> > /lib/libkmod.so.2.
> > And I have no idea where it pulled it from.
>
> > I checked MANIFEST files, d-i build log, even download all udebs in
> > bionic/bionic-updates,
> > extracted, but nothing shipped it.
>
> > This might be some d-i build magic, which I didn't look/find yet.
>
> You and me both! I hit the problem when I was making the test mini.iso and
> test vmlinuz / initrd.gz files. What I did to workaround it was do a grep over
> the local repository cache d-i keeps when building images, and then do a cheeky
> sed -i s/libkmod2/libkmod2-udeb/g, commands below:
>
> "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27),
> libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31),
> libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g"
> ./apt.udeb/state/lists/nz.archive.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages"
>
> "sed -i "s/Depends: libblkid1-udeb (>= 2.31), libc6-udeb (>= 2.27),
> libkmod2, util-linux-udeb/Depends: libblkid1-udeb (>= 2.31),
> libc6-udeb (>= 2.27), libkmod2-udeb, util-linux-udeb/g"
> ./apt.udeb/state/lists/security.ubuntu.com_ubuntu_dists_bionic-updates_main_debian-installer_binary-amd64_Packages"
>
> I then re-ran "fakeroot make build_monolithic". Apt seemed to be perfectly happy
> with the changes, even though libkmod2-udeb was missing from the archive, and
> produced the test install media correctly.
>
> I suppose that hack worked since libkmod2-udeb didn't actually ship
> any libraries,
> and /lib/libkmod.so.2 turns up through another magical mechanism.
>
> As for that magical mechanism, I found this in the README file in
> debian-installer/build/README:
>
> * Apt is used to download the required udebs. This does *not* include
>   libraries; libraries used by udebs must be installed on the build system,
>   and so are build-depended on.
>
> Sounds to me that if we include libkmod2, it copies the library from the build
> host into the install image. That's cheating =p
>
> As for the fix, well, I think the path of least resistance is to simply revert
> the change to libkmod2, and get it to start building libkmod2-udeb
> again (in bionic).
>
> Sure, it will incorrectly advertise itself as containing something useful when
> it really doesn't, but everything was fine for the two years it existed in the
> archive. I wouldn't revert it on anything beyond Bionic though.
>
> I'll send the above to Lukasz in case he has seen the dependency problem and
> had been looking into it.
>
> Thanks,
> Matthew
>
> On Sat, Jul 25, 2020 at 12:49 PM Mauricio Oliveira
> <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> >
> > Hi Matthew,
> >
> > Long story short, the debian-installer rebuild on PPA fails on Bionic.
> >
> > The fix for this should apparently involve changes to kmod and systemd,
> > at least systemd, and I think this may get push back from release team
> > at this time before the point release (however small the change is.)
> >
> > If the fix is needed only in the network-installer, that can be an SRU later.
> >
> > Let me explain, and hopefully you can start addressing it with Lukasƶ
> > on your Monday, before I join.
> >
> > The d-i build on PPA fail because 'udev-udeb' cannot pull 'libkmod2'.
> >
> >     Building dependency tree...
> >       Installing libslang2-udeb as Depends of cdebconf-newt-udeb
> >       udev-udeb:amd64 Depends on libkmod2:amd64 < none @un H > can't
> > be satisfied!
> >
> >     ...
> >
> >     The following packages have unmet dependencies:
> >      udev-udeb : Depends: libkmod2 but it is not installable
> >     E: Unable to correct problems, you have held broken packages.
> >     Makefile:620: recipe for target
> > 'stamps/get_udebs-cdrom_isolinux-stamp' failed
> >     make[7]: *** [stamps/get_udebs-cdrom_isolinux-stamp] Error 100
> >
> > This is correct, because the d-i build does configure sources only for udeb
> > packages, and libkmod2 is a regular deb.
> >
> > Previously it worked because udev-udeb pulled libkmod2-udeb (note suffix.)
> >
> > This is a regression in the dependencies of udev-udeb in 237-3ubuntu10.40:
> >
> >     https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.39
> >         Depends on: libkmod2-udeb
> >
> >     https://launchpad.net/ubuntu/bionic/amd64/udev-udeb/237-3ubuntu10.40
> >         Depends on: libkmod2
> >
> > This happened because the build dependency kmod (which you're familiar with :)
> > changed versions in between, 24-1ubuntu3.2 to 3.3:
> >
> >     On u10.39: Get:45 http://ftpmaster.internal/ubuntu
> > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.2 [40.0 kB]
> >     On u10.40: Get:45 http://ftpmaster.internal/ubuntu
> > bionic-updates/main amd64 libkmod2 amd64 24-1ubuntu3.3 [40.1 kB]
> >
> > And this change caused this:
> >
> >     kmod (24-1ubuntu3.3) bionic; urgency=medium
> >     ...
> >       * Removed --add-udeb from dh_makeshlibs, since libkmod2-udeb does not
> >         actually contain a library. (Closes: #939779)
> >     ...
> >
> > This still builds/ships the libkmod2-udeb package, but...
> >
> > The lack of dh_makeshlibs with --add-udeb causes the libkmod2
> > (non-udeb)'s "shlibs"
> > file to no longer indicate that udeb packages which depend on that
> > library package
> > should depend on the -udeb package instead (exactly what happens here.)
> >
> > From dh_makeshlibs(1):
> >
> >        --add-udeb=udeb
> >            Create an additional line for udebs in the shlibs file and
> > use udeb as the package name for udebs to depend on instead of the
> > regular library package.
> >
> > This can be confirmed with:
> >
> >     $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.2
> >     $ pull-lp-debs -a amd64 libkmod2 bionic 24-1ubuntu3.3
> >
> >     $ dpkg-deb --control libkmod2_24-1ubuntu3.2_amd64.deb libkmod2-32-control
> >     $ dpkg-deb --control libkmod2_24-1ubuntu3.3_amd64.deb libkmod2-33-control
> >
> >     $ cat libkmod2-32-control/shlibs
> >     libkmod 2 libkmod2
> >     udeb: libkmod 2 libkmod2-udeb
> >
> >     $ cat libkmod2-33-control/shlibs
> >     libkmod 2 libkmod2
> >
> > And according to systemd/debian/control file's entry for udev-udeb:
> >
> >     Package: udev-udeb
> >     ...
> >     Depends: ${shlibs:Depends},
> >          ${misc:Depends},
> >          util-linux-udeb
> >
> > That dependency comes from ${shlibs:Depends}, handled by dh_shlibdeps(1):
> >
> >     https://www.debian.org/doc/manuals/maint-guide/dreq.en.html
> >     """
> >     dh_shlibdeps(1) calculates shared library dependencies for binary packages.
> >     It generates a list of ELF executables and shared libraries it has
> > found for each binary package.
> >     This list is used for substituting ${shlibs:Depends}.
> >     """
> >
> > So,
> >
> > Since d-i needs both udev-udeb and libkmod2-udeb (for udev,
> > load/remove modules, etc)
> > fixing the udev-udeb dependency would require *at least* a
> > change/rebuild to systemd
> > to update the Depends: field of udev-udeb.
> >
> > Say, even if we came up with a hack/solution that doesn't need a
> > change/fix to libkmod2.
> >
> > This is where I think the SRU/release team might prefer to hold things
> > for release.
> > (systemd.)
> >
> > At this point I think that *if* the case/matter is only about the
> > network installer,
> > not the ISO (which is indeed critical with 18.04.5), then it can be
> > fixed via SRU,
> > without any concerns.
> >
> > Would you please evaluate this w/ customer/Lukasƶ on Monday?
> >
> >
> > A bit on libkmod2 change:
> >
> > The change/regression of --add-udeb to libkmod2 is not incorrect on
> > its own, since
> > libkmod2-udeb doesn't ship a shared library.
> >
> > I discussed this a bit with Rafael Tinoco (which introduced the
> > change), and perhaps
> > the change can be reverted (I'd have to check with a rebuild of kmod
> > w/ it reverted.)
> >
> > (then rebuild kmod, then systemd, then d-i -- after debootstrap w/
> > your fix, of course.
> > See it's a chain that looks big for this time before the release. :)
> >
> >
> > Some tech details:
> >
> > udev-udeb depends on libkmod2 shared library because of udevadm, for
> > the kmod built-in.
> >
> > You may notice that even before the --add-udeb flag removal,
> > libkmod2-udeb *did not*
> > ship that shared library!
> >
> > I wondered how the installer worked. It DOES have that library in
> > /lib/libkmod.so.2.
> > And I have no idea where it pulled it from.
> >
> > I checked MANIFEST files, d-i build log, even download all udebs in
> > bionic/bionic-updates,
> > extracted, but nothing shipped it.
> >
> > This might be some d-i build magic, which I didn't look/find yet.
> >
> >
> > Finally, I'm a bit tired to think now, so I'll give you the ball for now to play
> > a bit on considerations of 1) should this really be fixed before
> > release for ISO,
> > or just network installer is OK?   and more fun in 2) how to properly
> > fix this :)
> >
> > Thanks and have a great weekend!
> >
> > P.S.: I'm attaching my hacky notes about this today, in case it might help.
> >
> > On Fri, Jul 24, 2020 at 7:27 PM Mauricio Oliveira
> > <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> > >
> > > Matthew,
> > >
> > > By the way, per the SF case, the customer uses the network install method.
> > > This can be fixed post-release, as SRUs to debian-installer will
> > > produce new netboot images.
> > >
> > > Hopefully they (and other people) don't plan to try to use HTTPS on
> > > the _ISO_ installer?
> > > The ISO image is what is more strictly tied to the release date.
> > > I'm not sure how much that scenario makes sense.
> > > (there may be the case of trying to use HTTPS for the ISO installer to
> > > download updates though?)
> > >
> > > I'm saying this because this is now (of course!) heading into systemd :-)
> > > specifically the dependencies of the udev-udeb package.  And that may
> > > be just an adventure
> > > to try to fix / SRU before release date. :)
> > >
> > > I'll give more details later.
> > >
> > > cheers,
> > >
> > >
> > > On Fri, Jul 24, 2020 at 6:30 PM Mauricio Oliveira
> > > <mauricio.oliveira@xxxxxxxxxxxxx> wrote:
> > > >
> > > > Definitely looking at it right now. :)
> > > >
> > > > The debootstrap portion is OK. However,
> > > >
> > > > there's a PPA build error with the required rebuild of d-i in Bionic
> > > > (not on Focal -- there's always fun on the releases we actually need!)
> > > > due to udev-udeb changing dependencies from libkmod2-udeb to libkmod2
> > > > (non-udeb), which I just tracked down to kmod 24-1ubuntu3.3.
> > > >
> > > > I'll provide a write up on the LP bug (and mention internally in SF
> > > > case) and hopefully it should be relatively simple to fix.
> > > >
> > > > As mentioned, debootstrap looks good; so I'll move that one soon.
> > > >
> > > > cheers,
> > > >
> > > > On Fri, Jul 24, 2020 at 6:25 PM Dan Streetman
> > > > <dan.streetman@xxxxxxxxxxxxx> wrote:
> > > > >
> > > > > Hi Matthew,
> > > > >
> > > > > As @mfo has done work in this (installer image) area before (LP:
> > > > > #1807023) I asked him to take a look; I'm cc'ing our sts sponsors
> > > > > email list (which includes Mauricio) also.
> > > > >
> > > > > I think he's handling the sponsoring, so hopefully it'll make it for
> > > > > the point release!
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Fri, Jul 24, 2020 at 1:14 AM Matthew Ruffell
> > > > > <matthew.ruffell@xxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > > Hi Dan and Eric,
> > > > > >
> > > > > > [URGENT] Please review and sponsor LP1772556. This needs to make 18.04.5 LTS.
> > > > > >
> > > > > > https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1772556
> > > > > >
> > > > > > I have written to Lukasz Zemczak and made him aware of this bug, and I have also
> > > > > > asked him to review and sponsor the debdiffs, since he was the last to modify
> > > > > > debootstrap in Bionic, and is also in charge of producing 18.04.5 LTS images.
> > > > > >
> > > > > > I haven't heard back from Lukasz though. I emailed him on my Wednesday, and
> > > > > > on my Thursday.
> > > > > >
> > > > > > Could you please maybe ping Lukasz during your day to make sure he got my email?
> > > > > >
> > > > > > The debdiffs in the LP bug are kinda weird. debootstrap uses "3.0 (native)"
> > > > > > instead of "3.0 (quilt)", and the debdiff seems to take diffs of all the ubuntu
> > > > > > releases, even though they are all symlinks to a single file that got changed,
> > > > > > which is "gutsy".
> > > > > >
> > > > > > Anyway, please nudge Lukasz, or sponsor my debdiffs. Konica Minolta filed a
> > > > > > L2 about this in SF289200, and it needs to make 18.04.5 LTS or it will never be
> > > > > > fixed.
> > > > > >
> > > > > > Thanks!
> > > > > > Matthew
> > > > >
> > > > > --
> > > > > Mailing list: https://launchpad.net/~sts-sponsors
> > > > > Post to     : sts-sponsors@xxxxxxxxxxxxxxxxxxx
> > > > > Unsubscribe : https://launchpad.net/~sts-sponsors
> > > > > More help   : https://help.launchpad.net/ListHelp
> > > >
> > > >
> > > >
> > > > --
> > > > Mauricio Faria de Oliveira
> > >
> > >
> > >
> > > --
> > > Mauricio Faria de Oliveira
> >
> >
> >
> > --
> > Mauricio Faria de Oliveira



-- 
Mauricio Faria de Oliveira