cross-toolchain-base-devs team mailing list archive

[Helmut Grohne <helmut@xxxxxxxxxx>]

Hi,

In this mail, I'll try to summarize my state of knowledge on this
matter while noting that I don't see the full picture.

On Fri, Mar 29, 2024 at 11:17:55AM +0100, Bastian Blank wrote:
> On Thu, Mar 21, 2024 at 08:48:01PM +0100, Helmut Grohne wrote:
> > I was recently working on gcc builds and this disagreement currently
> > makes stuff unbuildable. Hence I looked into solutions and/or
> > workarounds.
> 
> Care to just share what you actually found?  Where is it broken and how
> to see this?
> 
> Because this whole thing started with "it is broken, but I won't tell
> you where or what or how".

Quite clearly, this is not a single problem, but a mesh of problems and
in a few cases it is not obvious where to solve them.

> > As a result, I implemented the proposed change and am attaching it for
> > discussion here. I've implemented it in a way that if there is a sysroot
> > linux header installation, it'll be preferred. Do you see any downsides
> > of this approach?
> 
> I wonder now.  How would that ever work for the native build?  Or does
> the native build already do those symlinks?  Or are native and cross
> configured differently?  Or is that a weird difference in gcc itself?

Native and cross builds work quite differently indeed.

So let me first try to collect all relevant problems that I encountered
here. I vaguely try to list the more important ones earlier. I have
caused some of these problems and don't want to assign any blame but
look for solutions.

1. API expectation of *-$arch-cross packages

   When *-$arch-cross packages were first introduced before multiarch
   was a thing, there was (and still is) and implicit contract saying
   that their functionality is to be provided within the
   /usr/$DEB_HOST_GNU_TYPE hierarchy. In particular, this layout does
   not interfere with multiarch on a filesystem level and hence
   *-$arch-cross packages typically are arch:all m-a:foreign.

   In particular, linux-libc-dev now provides such packages without
   actually providing those paths violating this implicit contract.

2. Violation of Multi-Arch: foreign

   linux-libc-dev was changed from an Arch:any + Multi-Arch:same package
   to an Architecture:all + Multi-Arch:foreign package. It does so by
   providing the headers for all architectures in a single package via
   symlink farms. "all" architectures is debatable though. The set of
   architectures changes rather frequently with new ones being added and
   old ones being removed. Therefore, linux-libc-dev will often look
   like it provides linux-libc-dev:$somearch to dependency resolvers
   while in fact not providing the functionality - thus violating the
   promise of Multi-Arch: foreign. For example, linux-libc-dev is
   currently dysfunctional for arc, but next year it'll be a different
   architecture.

   Moreover, looking at the metadata of linux-libc-dev initially did not
   provide means of figuring out which architectures were actually
   supported and which were not. This has been changed as linux-libc-dev
   now Provides linux-libc-dev-$arch-cross packages (causing the first
   problem), but at least giving bootstrappers a means to figure out
   whether a given linux-libc-dev package is usable to them.

3. linux-libc-dev consumes much space on mirrors and installations

   linux-libc-dev originally was Arch:any and yet much of its content
   was the same across architectures albeit in different paths. Thus,
   the size of the .deb (multiplied by the number of architectures) was
   quite big and also coinstalling linux-libc-dev would result in
   duplicate files being extracted to multiple locations increasing the
   installation size in a multiarch setting.

   As a result, linux-libc-dev now is Arch:all and we only get to have
   one package. It grew from about 1.8MB (times 10 architectures) to
   about 2.2MB and its installed size grew from about 7MB (per
   architecture) to 10MB (for all architectures).

   This change caused the second problem.

4. cross-toolchain-base being bd-uninstallable

   cross-toolchain-base cannot currently be built (FTBFS #1064003 and
   #1067370) and one of the aspects is that it declares Build-Conflicts
   with linux-libc-dev-$arch-cross. The recently added Provides on
   linux-libc-dev satisfy them and thus cause cross-toolchain-base to be
   bd-uninstallable.

   I don't exactly understand why it declares them, but I guess that
   having a different set of kernel headers available in
   /usr/$DEB_HOST_GNU_TYPE would cause them to be picked up by the build
   and potentially cause misbuilds. cross-toolchain-base builds these
   packages and it also uses them during build of glibc.

5. gcc-V-cross not being buildable

   The gcc-V-cross package relies on -$arch-cross packages usually built
   from cross-toolchain-base and expects them to provide their
   functionality in /usr/$DEB_HOST_GNU_TYPE. The current linux-libc-dev
   provides the package names but not the expected path (this actually
   is the first problem) and as a consequence, gcc-V-cross currently
   fails to build from source.

6. Cross bootstrap cannot tell whether linux-libc-dev supports an
   architecture

   Before linux-libc-dev gained the -$arch-cross Provides, there was no
   way of determining whether linux-libc-dev supports an architecture by
   looking at its metadata. As a result, cross bootstrap had now way of
   knowing whether linux-libc-dev needed to be rebuilt. This is a
   consequence of the second problem.

7. Cross bootstrap needs to deal with Arch:all packages

   Until linux-libc-dev became an architecture-dependent Arch:all
   package, the entire cross bootstrap was possible by only performing
   arch-only builds and using a repository of only arch:any packages
   used in conjunction with a Debian mirror restricted to Arch:all
   packages. Now, the bootstrap repository must also be able to carry
   an Arch:all package and handle the fact that multiple versions of
   linux-libc-dev exist in a bootstrap setting one of which does not
   work (as a result of the second problem).

8. Duplication of functionality via -$arch-cross packages

   When performing cross builds, we currently need both -$arch-cross
   packages and :$arch packages for glibc and linux-libc-dev. These can
   be built from different versions and we know that using
   libc6-dev-$arch-cross packages built from a different glibc version
   than libc6-dev:$arch together causes problems (repeatedly) and hence
   glibc now declares Conflicts with old libc6-dev-$arch-cross packages.
   If gcc-V-cross were to use :$arch packages, it would have to declare
   cross-architecture dependencies, which is not currently supported by
   our buildd network. That is one reason for it still using
   -$arch-cross packages.

When I say problems here, you may read that more as "challenge" and in
particular, it is not obvious that the change that caused a problem
should be reverted. Rather, there are often multiple ways to resolve
these. Let me sketch a few options that may not be obvious from this
list:

1+3+5. linux-libc-dev could add a symlink farm for
   /usr/$DEB_HOST_GNU_TYPE.

1+3+4+6. linux-libc-dev could rename its Provides dropping the -cross
   suffix.

2. Ignore the violation. libc6 also violates Multi-Arch:same, because
   the dynamic loader conflicts for some pairs of architectures. There
   is nothing that can be done about this as architecture-dependent
   breaks or conflicts are not supported. Hence, we can argue that this
   violation is so useful that it should be tolerated.

2+3+6+7. linux-libc-dev could be split into linux-libc-dev-common
   arch:all m-a:foreign and the symlink farms could be kept in
   linux-libc-dev:any m-a:same retaining the size reduction.

3. cross-toolchain-base could build a linux-libc-dev-cross package
   that Provides the earlier -$arch-cross packages and contains a
   similar symlink farm to linux-libc-dev.

3+8. cross-toolchain-base could stop creating linux-libc-dev-$arch-cross
   packages.
   Patch: https://bugs.debian.org/1059786#10

4. cross-toolchain-base could drop its Build-Conflicts. This may cause
   further problems though.
   Patch: https://bugs.debian.org/1067370#17

5. gcc-V-cross could create its own symlink farm for linux-libc-dev at
   build time and pass that to the gcc build removing the need for the
   /usr/$DEB_HOST_GNU_TYPE location.
   Patch: https://bugs.debian.org/1065416#129

7. cross bootstrap can implement indep builds.

8. Implement cross-architecture dependencies in the buildd network such
   that we can get rid of -$arch-cross packages.

Please consider both the list of problems and in particular the list of
options incomplete.

On a procedural level, linux-libc-dev hijacked
linux-libc-dev-$arch-cross. After I sent
https://bugs.debian.org/1059786#10, there was quite some silence which I
mistakenly interpreted as agreement and hence suggested to Bastian that
linux-libc-dev should provide linux-libc-dev-$arch-cross as a solution
to the sixth problem.

Regarding CTTE, I am prejudiced on this matter and want to abstain from
voting.

Helmut