cross-toolchain-base-devs team mailing list archive

[Helmut Grohne <helmut@xxxxxxxxxx>]

Source: cross-toolchain-base
Version: 45
Severity: wishlist
Tags: patch moreinfo

I'm not sure anymore who told me, but glibc has a build_many_glibcs.py
script and it does the toolchain bootstrap dance with fewer stages than
Debian (i.e. cross-toolchain-base and rebootstrap) does. The current
Debian toolchain bootstrap looks like this:

 * binutils
 * linux-libc-dev
 * gcc stage1 (bare metal)
 * glibc stage1 (headers + stub libc.so)
 * gcc stage2
 * glibc stage2 (libc without systemtap and other optional features)
 * gcc stage3 (libc-integrated cross compiler)
 * gcc rtlibs (runtime libraries for the cross compiler)

The assertion is that we can skip glibc stage1 and gcc stage2 and go
directly from gcc stage1 to glibc stage2.

I've implemented this in rebootstrap and it seems to work reasonably
well for a number of architectures. I've not run it on the full test
matrix yet. Some observations on rebootstrap:
 * musl-linux-any already works like this since a while.
 * hurd-any formerly needed libihash-dev for libpthread, but no longer
   does. This sequence also works on hurd-i386.
 * I've had success with arm64, armel, armhf, m68k, mips (nobiarch),
   mipsel (nobiarch) and ppc64el thus far. Notably, these all lack
   multilibs and I'm still sorting out multilib issues.
 * I cannot tell about kfreebsd-any, since they didn't manage to get the
   relevant kernel header source package back into unstable yet.

I've implemented this for cross-toolchain-base (patch attached) and a
performed a successful testbuild. Using diffoscope to compare the
resulting packages with the ones from the archive was not a useful thing
to do as the build-ids changed.

So what do you think about going to fewer stages? I'd like to solicit
explicit feedback from the involved parties:

gcc maintainers / Matthias:
 * Do you have any objections to reducing stages?
 * Should we delete gcc stage2?
 * Should we rename gcc stage3 to gcc stage2?

glibc maintainers / Aurelien:
 * Do you have any objections to reducing stages?
 * Should we delete glibc stage1?
 * Should we rename glibc stage2 to glibc stage1?
 * Should we maybe split stage2 into a number of profiles
   pkg.glibc.noselinux, pkg.glibc.noxen, pkg.glibc.noalphaev,
   pkg.glibc.noxcryptdep?

Due to these open questions, I've tagged the bug moreinfo.

Helmut

diff --minimal -Nru cross-toolchain-base-45/debian/changelog cross-toolchain-base-45+nmu1/debian/changelog
--- cross-toolchain-base-45/debian/changelog	2020-03-22 14:02:54.000000000 +0100
+++ cross-toolchain-base-45+nmu1/debian/changelog	2020-04-15 11:35:18.000000000 +0200
@@ -1,3 +1,10 @@
+cross-toolchain-base (45+nmu1) UNRELEASED; urgency=medium
+
+  * Non-maintainer upload.
+  * Bootstrap with fewer stages. (Closes: #-1)
+
+ -- Helmut Grohne <helmut@xxxxxxxxxx>  Wed, 15 Apr 2020 11:35:18 +0200
+
 cross-toolchain-base (45) unstable; urgency=medium
 
   * Build using glibc 2.30-2.
diff --minimal -Nru cross-toolchain-base-45/debian/rules cross-toolchain-base-45+nmu1/debian/rules
--- cross-toolchain-base-45/debian/rules	2020-03-22 14:02:01.000000000 +0100
+++ cross-toolchain-base-45+nmu1/debian/rules	2020-04-15 11:35:18.000000000 +0200
@@ -343,96 +344,6 @@
 	  ln -sf ${CROSS_GNU_TYPE}-gcov-${VER_GCC_BASE} ${CROSS_GNU_TYPE}-gcov
 endef
 
-$(stamp)build-gcc2.%: $(stamp)init-gcc $(stamp)install-glibc1.%
-	@echo START $@
-	cd debian/tmp.${CROSS_ARCH}/$(PF)/bin/ && \
-	  rm -f ${CROSS_GNU_TYPE}-gcc-${VER_GCC_BASE} ${CROSS_GNU_TYPE}-gcc && \
-	  rm -f ${CROSS_GNU_TYPE}-cpp-${VER_GCC_BASE} ${CROSS_GNU_TYPE}-cpp && \
-	  rm -f ${CROSS_GNU_TYPE}-gcov-${VER_GCC_BASE} ${CROSS_GNU_TYPE}-gcov
-	cd gcc && \
-	  PATH=${CURDIR}/debian/tmp.${CROSS_ARCH}/$(PF)/bin:${PATH} \
-	  LD_LIBRARY_PATH=$(call binutils_ldpath,$*):${CURDIR}/debian/tmp.${CROSS_ARCH}/usr/lib:${CURDIR}/debian/tmp.${CROSS_ARCH}/lib \
-	  DH_VERBOSE=1 \
-	  WITH_SYSROOT=/ \
-	  WITH_BUILD_SYSROOT=${CURDIR}/debian/tmp.${CROSS_ARCH} \
-	  DEB_STAGE=stage2 \
-	  PKG_IGNORE_CURRENTLY_BUILDING=1 \
-	  BACKPORT=false \
-	  DEB_BUILD_OPTIONS="$(DEB_BUILD_OPTIONS) nocheck nopgo nolto nohppa64 nojit nonvptx" \
-	  WITHOUT_LANG="hppa64 jit nvptx" \
-	  $(if $(filter $(HOST_ARCH),$(CROSS_ARCH)),FORCE_CROSS_LAYOUT=yes WITH_BOOTSTRAP=off) \
-	  dpkg-buildpackage -b -uc -us -d
-	touch $@
-
-$(stamp)install-gcc2.%: $(stamp)build-gcc2.%
-	@echo START $@
-	$(call install_gcc)
-	dpkg-deb -x libgcc[124]-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}
-ifneq (,$(ARM32_MULTILIBS))
-	$(if $(filter $(CROSS_ARCH),armhf), \
-	  dpkg-deb -x libsfgcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libsfgcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),armel), \
-	  dpkg-deb -x libhfgcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libhfgcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-endif
-	$(if $(filter $(CROSS_ARCH),amd64 ppc64 s390x sparc64 x32), \
-	  dpkg-deb -x lib32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),i386 powerpc x32), \
-	  dpkg-deb -x lib64gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib64gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),amd64 i386), \
-	  dpkg-deb -x libx32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libx32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mips mipsel mipsr6 mipsr6el), \
-	  dpkg-deb -x lib64gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib64gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libn32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libn32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mipsn32 mipsn32el mipsn32r6 mipsn32r6el), \
-	  dpkg-deb -x lib64gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib64gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mips64 mips64el mips64r6 mips64r6el), \
-	  dpkg-deb -x lib32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x lib32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libn32gcc1-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libn32gcc-${VER_GCC_BASE}-dev-${CROSS_ARCH}-cross_${DEB_VER_GCC}_all.deb \
-		debian/tmp.${CROSS_ARCH} \
-	)
-	touch $@
-
 $(stamp)build-gcc3.%: $(stamp)init-gcc $(stamp)install-glibc2.%
 	@echo START $@
 	cd debian/tmp.${CROSS_ARCH}/$(PF)/bin/ && \
@@ -468,88 +379,7 @@
 #	fi
 	touch $@
 
-$(stamp)build-glibc1.%: $(stamp)init-glibc $(stamp)install-linux.% $(stamp)install-gcc1.%
-	@echo START $@
-# FIXME: why debian/libgcc1-armel-cross?
-	cd glibc-${VER_GLIBC} && \
-	  PATH=${CURDIR}/debian/tmp.${CROSS_ARCH}/$(PF)/bin/:${PATH} \
-	  LD_LIBRARY_PATH=$(call binutils_ldpath,$*):${CURDIR}/gcc/debian/libgcc1-armel-cross/usr/${CROSS_GNU_TYPE}/lib/ \
-	  LINUX_SOURCE=${CURDIR}/debian/tmp.${CROSS_ARCH}/usr \
-	  PATCHED_SOURCES=yes \
-	  WITH_BUILD_SYSROOT=${CURDIR}/debian/tmp.${CROSS_ARCH} \
-	  DEB_BUILD_PROFILES=stage1 \
-	  PKG_IGNORE_CURRENTLY_BUILDING=1 \
-	  BACKPORT=false \
-	  DEB_BUILD_OPTIONS="$(DEB_BUILD_OPTIONS) nocheck" \
-	  CTB_LIBC_DEV_DEPENDS='$(CTB_LIBC_DEV_DEPENDS)' \
-	  fakeroot dpkg-buildpackage -B -uc -us -a${CROSS_ARCH} -d -Pstage1
-	: # FIXME: a libc6.1-alphaev67 package is built on alpha
-	$(if $(filter $(CROSS_ARCH),alpha),  rm -f libc6.1-alphaev67*.deb)
-	touch $@
-
-$(stamp)install-glibc1.%: $(stamp)build-glibc1.%
-	@echo START $@
-	$(if $(filter $(CROSS_ARCH),alpha ia64), \
-	  dpkg-deb -x libc6.1-dev_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}, \
-	  dpkg-deb -x libc6-dev_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-
-ifneq (,$(ARM32_MULTILIBS))
-	$(if $(filter $(CROSS_ARCH),armhf), \
-	  dpkg-deb -x libc6-dev-armel_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),armel), \
-	  dpkg-deb -x libc6-dev-armhf_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-endif
-	$(if $(filter $(CROSS_ARCH),ppc64), \
-	  dpkg-deb -x libc6-dev-powerpc_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),s390x), \
-	  dpkg-deb -x libc6-dev-s390_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),sparc64), \
-	  dpkg-deb -x libc6-dev-sparc_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),powerpc), \
-	  dpkg-deb -x libc6-dev-ppc64_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mips mipsel mipsr6 mipsr6el), \
-	  dpkg-deb -x libc6-dev-mips64_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-mipsn32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mipsn32 mipsn32el mipsn32r6 mipsn32r6el), \
-	  dpkg-deb -x libc6-dev-mips64_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-mips32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),mips64 mips64el mips64r6 mips64r6el), \
-	  dpkg-deb -x libc6-dev-mips32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-mipsn32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),amd64), \
-	  dpkg-deb -x libc6-dev-i386_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-x32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),i386), \
-	  dpkg-deb -x libc6-dev-amd64_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-x32_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	$(if $(filter $(CROSS_ARCH),x32), \
-	  dpkg-deb -x libc6-dev-amd64_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH}; \
-	  dpkg-deb -x libc6-dev-i386_${DEB_VER_GLIBC}_${CROSS_ARCH}.deb debian/tmp.${CROSS_ARCH} \
-	)
-	: # FIXME: why is this needed?
-	cd debian/tmp.${CROSS_ARCH}/usr/lib ;\
-	for crt in ${CROSS_GNU_TYPE}/crt*; do \
-		ln -sf $$crt `basename $$crt`; \
-	done
-	: # dpkg-shlibdeps for the stage2 gcc needs these
-	for d in $$(find debian/tmp.${CROSS_ARCH}/usr/lib -name libc.so -printf '%h\n'); do \
-	  ln -sf libc.so $$d/libc.so.6; \
-	done
-	touch $@
-
-$(stamp)build-glibc2.%: $(stamp)init-glibc $(stamp)install-gcc2.%
+$(stamp)build-glibc2.%: $(stamp)init-glibc $(stamp)install-linux.% $(stamp)install-gcc1.%
 	@echo START $@
 # FIXME: why debian/libgcc1-armel-cross?
 	cd glibc-${VER_GLIBC} && \
@@ -1343,12 +1173,8 @@
 	$(stamp)init-gcc \
 	$(stamp)build-gcc1.% \
 	$(stamp)install-gcc1.% \
-	$(stamp)build-gcc2.% \
-	$(stamp)install-gcc2.% \
 	$(stamp)build-gcc3.% \
 	$(stamp)init-glibc \
-	$(stamp)build-glibc1.% \
-	$(stamp)install-glibc1.% \
 	$(stamp)build-glibc2.% \
 	$(stamp)install-glibc2.% \
 	$(stamp)debian-files-base \