launchpad-reviewers team mailing list archive

[William Grant <me@xxxxxxxxxxxxxxxxxx>]

William Grant has proposed merging lp:~wgrant/launchpad/remove-obsolete-buildd-junk into lp:launchpad/devel.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)


This branch removes obsolete junk from launchpad-buildd. Deleted are some ancient preliminary build farm design docs and example.chroot, an example configuration file for buildd-slave-chroot-tool, which was itself removed some time ago.
-- 
https://code.launchpad.net/~wgrant/launchpad/remove-obsolete-buildd-junk/+merge/30233
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/remove-obsolete-buildd-junk into lp:launchpad/devel.

=== modified file 'lib/canonical/buildd/debian/rules'
--- lib/canonical/buildd/debian/rules	2010-06-12 03:48:34 +0000
+++ lib/canonical/buildd/debian/rules	2010-07-19 04:56:00 +0000
@@ -60,7 +60,6 @@
 	install -m755 buildd-config.py $(target)/usr/bin/buildd-genconfig
 	install -m644 sbuildrc $(targetshare)/sbuildrc
 	install -m644 sbuild.conf $(target)/etc/sbuild.conf
-	install -m644 example.chroot debian/launchpad-buildd/usr/share/doc/launchpad-buildd/
 	install -m755 debian/upgrade-config $(targetshare)/upgrade-config
 	install -m755 check-implicit-pointer-functions $(target)/usr/bin/check-implicit-pointer-functions
 	# Okay, that's installed all the slave-related files

=== removed file 'lib/canonical/buildd/example.chroot'
--- lib/canonical/buildd/example.chroot	2009-12-16 00:16:24 +0000
+++ lib/canonical/buildd/example.chroot	1970-01-01 00:00:00 +0000
@@ -1,46 +0,0 @@
-# Example chroot configuration file
-
-[chroot]
-# What distroarchrelease are we after
-distribution=ubuntu
-distroseries=breezy
-# Specify "host" to build for this host, otherwise try specifying an archtag
-architecture=host
-
-# We usually want the buildd variant
-variant=buildd
-
-# Where does debootstrap find the archive?
-debootstraparchive=http://archive.ubuntu.com/ubuntu
-
-[apt]
-# What is the archive root?
-archiveroot=http://dogfood.internal/ubuntu
-# What components do we enable?
-components=main restricted universe
-# What pockets (other than the release pocket?)
-#pockets=security updates
-pockets=
-
-# Disabling the verification is handy because it prevents issues
-# where the signing key changes or the signatures are not available
-signingdisabled=1
-
-[buildd]
-# Information for setting up the buildd configuration inside the chroot
-user=buildd
-group=buildd
-paths=/build /build/buildd /home/buildd /var/debbuild /var/debbuild/srcdep-lock
-touchfiles=/var/debbuild/avg-build-times /var/debbuild/avg-build-space /CurrentlyBuilding
-
-# Extra packages we should install for the buildd
-extrapkgs=build-essential fakeroot
-
-[intervention]
-# At what stages would you like to intervene in the process?
-postbootstrap=0
-postextrapkgs=0
-postfinalapt=0
-# Currently this is directly after postfinalapt
-pretarup=1
-

=== removed file 'lib/canonical/buildd/notes.txt'
--- lib/canonical/buildd/notes.txt	2007-10-31 19:03:15 +0000
+++ lib/canonical/buildd/notes.txt	1970-01-01 00:00:00 +0000
@@ -1,206 +0,0 @@
-Launchpad Build Infrastructure
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Points of concern currently...
-
-SourcePackageRelease records may not be associated with any specific
-DistroSeries. If that happens to be the case; what DistroArchSeries
-do we build them for?
-
-
-Database changes
-^^^^^^^^^^^^^^^^
-
-CREATE TABLE BuildQueue (
-  id SERIAL PRIMARY KEY,
-  build INTEGER FOREIGN KEY REFERENCES build(id) NOT NULL,
-  builder INTEGER FOREIGN KEY REFERENCES builder(id),
-  created TIMESTAMP WITH TIMEZONE NOT NULL,
-  buildstart TIMESTAMP WITH TIMEZONE,
-  logtail TEXT,
-
-);
-
--- If speedindex is NULL, we don't know, else it is the number of seconds
--- needed to build a reference package or something.
-ALTER TABLE Builder ADD COLUMN speedindex INTEGER;
-
-Basic case decision on what to build
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For each DistroSeries which is currently marked for
-autobuilding... Find all SourcePackageReleases published into that
-DistroSeries for which the set of Build records does not contain
-entirely the set of DistroArchSeries being built for that
-DistroSeries.
-
-For each of those; create a build record in "Needs Building" state
-
-Next gather every Build record in the "Needs Building" state for which
-no BuildQueue entry exists and create the BuildQueue entry.
-
-For each entry in the BuildQueue for whom the Build record states a
-builder, query the builder for that build's state.
-
- * complete successful builds are extracted; recorded; the BuildQueue
-   entry removed. The time to build is recorded for future estimates.
- * failed builds are logged as such, the Build record is marked as 
-   "Failed to build" and the BuildQueue entry is removed.
- * failed builds (failing due to dependencies) have their logs recorded
-   and the BuildQueue.builder is cleared; the Build is placed into
-   "Manual Dep Wait" state
- * failed builds (failing due to chroot issues) have the same
-   done for them and are placed into "Chroot Wait" state. Also a
-   chroot-damage mail is sent out.
- * failed builds (failing due to disk issues) are simply returned to
-   the "Needs Building" state, the builder is marked as failed and
-   a builder-failure mail is sent out. If that was the last builder
-   for a given architecture then lots of panicy emails are sent.
- * incomplete builds have their logtails updated.
-
-For each entry in the BuildQueue for whom the Build record states
-"Needs Building" (rather than ex. "Manual Dep Wait")
-
- * Calculate for each entry the chroot id needed.
- * Any entry for which the chroot is unavailable is placed into
-   "Chroot Wait" state and is pared from the list
- * Any entry for which the chroot is known bad is placed into the
-   "Chroot Wait" state
- * Any entry for which there is currently no available builder is
-   pared from the list.
- * For each remaining entry; the build-dependencies are analysed out
-   and a decision is made about whether or not the build is likely
-   to dep-wait or not. Any for which a dep-wait is inevitable is pared
-   out and left as "Needs Building"
- * For each remaining entry a score is calculated. This is based upon
-   the score generated by the Distro{,Release}'s policy engine for
-   deciding what order things get built. Also taken into account are
-   things like whether or not the sourcepackage has been built on that
-   DistroSeries before.
-   Also an estimation of the time to build the package is calculated.
-   This is measured as a fraction/multiple of the time it takes to
-   build a "standard" package. This combined with the speed measure
-   of each builder available for the architecture provides us with
-   an estimate of how long the package will take to build on any given
-   builder.
- * For each architecture in turn, order the pending builds by their
-   score. For each builder except the last; choose the next highest
-   scoring package and queue the start of the build on the builder.
-   For the last builder; reorder by estimated time to build (in an
-   exponential growth grouping) and by score. Queue the highest scoring
-   package in the shortest-time-to-build group.
- * For each queued action; download the required chroot to the builder
-   if it doesn't already have it; download the sources to it and then
-   initiate the build. Record the fact that it is building.
-
-The scoring mechanism for pending builds
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
- * Estimated timeto run (smaller == more points)
- * Time spent in the build queue (longer == more points)
- * Position in build dependency chains (depended on == more points)
- * Distribution specific policy (E.g. toolchains first)
- * Priority of package (Base before Desktop before Supported)
- * Urgency of upload
- * Overarching policy (E.g. security builds first)
-
-Time groupings for packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
- * Five minutes or less
- * Five to fifteen minutes
- * Fifteen to thirty minutes
- * Thirty to sixty minutes
- * Greater than sixty minutes
-
-
-The build-master <-> build slave protocol
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Operated entirely over XML-RPC... Always initiated by the master to
-the slave; the only outgoing sockets the slave is permitted are direct
-to the public archive machine.
-
-Buildd slaves have HTTP read-only access to librarian.
-
-Buildd master is called Fiera
-Buildd slaves are called Bob
-
-Calls that the master can make to the slave
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
- INFO (none) ->
-    Slave Protocol Version
-    [ Commands Understood ]    
-
- STATUS (none) ->
-    IDLE / BUILDING / WAITING / ABORTED
-    if BUILDING
-       BUILD-ID in progress
-       LOGTAIL (last 1kB or so of the log)
-    if WAITING
-       OK / DEPFAIL / PACKAGEFAIL / CHROOTFAIL / BUILDERFAIL
-       BUILD-ID waiting
-       List of filename <-> SHA1sum waiting for upload
-                if OK / PACKAGEFAIL
-    if ABORTED
-       BUILD-ID Aborted
-
- FETCHLOGTAIL (amount) ->
-    tail of log file (all of log file if amount <= 0) else (up-to) amount bytes
-    Returns an empty string if no build is in progress
-
- DOYOUHAVE (SHA1sum) ->
-    YES / NO  
-    (May fetch from the librarian and return YES if it lacks the file)
-
- STOREFILE (CONTENT) ->
-    Stores the CONTENT and returns SHA1sum
-
- FETCHFILE (SHA1sum) ->
-    CONTENT for that SHA1summed file
-
- STARTBUILD (BUILD-ID, SHA1sum <-> filename list, CHROOT SHA1sum,
-	     BUILDCOMMAND) ->
-    UNKNOWN-SUM 
-      SHA1sum
-    UNKNOWN-BUILDER
-    INITIATED
-
- ABORT (none) ->
-    Any build in progress is aborted and the builder is placed in
-    ABORTED, returns ABORTED
-
- CLEAN (none) ->
-    The file-cache gets cleaned up, state is reverted to IDLE
-    returns IDLE
-
-An example interaction
-^^^^^^^^^^^^^^^^^^^^^^
-
-STATUS() -> IDLE
-DOYOUHAVE(chrootsum) -> YES
-DOYOUHAVE(dscsum) -> NO
-STOREFILE(foo.dsc) -> dscsum
-DOYOUHAVE(origsum) -> YES
-DOYOUHAVE(diffsum) -> NO
-STOREFILE(foo.diff.gz) -> diffsum
-STARTBUILD(10023, { foo.dsc: dscsum,
-                    foo.orig.tar: origsum, 
-                    foo.diff.gz: diffsum },
-           chrootsum, DEBIAN-SOURCE) -> INITIATED
-STATUS() -> BUILDING, 10023, "...logtail"
-
-...time passes...
-
-STATUS() -> BUILDING, 10023, "...logtail"
-
-...time passes...
-
-STATUS() -> WAITING, OK, 10023, foo.deb=debsum, foo.changes=changessum,
-            BUILDLOG=logsum
-FETCHFILE(debsum) -> foo.deb
-FETCHFILE(changessum) -> foo.changes
-FETCHFILE(logsum) -> BUILDLOG
-CLEAN() -> OK
-STATUS() -> IDLE

=== removed file 'lib/canonical/buildd/sbuild-notes.txt'
--- lib/canonical/buildd/sbuild-notes.txt	2009-12-16 00:16:24 +0000
+++ lib/canonical/buildd/sbuild-notes.txt	1970-01-01 00:00:00 +0000
@@ -1,49 +0,0 @@
-Notes on how to handle sbuild
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The buildd slave runs as the user buildd (ensure UID=1003, due 'tar'
-silly defect when decompressing chroots)
-
-This user has sudo privs for running the chroots (must be passwordless)
-
-Sbuild needs patching:
-
-$main::build_source = 0;
-+#$main::jobs_file = "$main::HOME/build/build-progress";
--$main::jobs_file = "build-progress";
-$main::max_lock_trys = 120;
-
-  cleanup_close:
-	analyze_fail_stage( $pkgv );
-+	if( $main::pkg_status eq 'failed' ) {
-+	    $main::pkg_status = 'failed ' . $main::pkg_fail_stage;
-+	}
-	write_jobs_file( $main::pkg_status );
-
- close_log();
--unlink( $main::jobs_file ) if $main::batchmode;
-+#unlink( $main::jobs_file ) if $main::batchmode;
- unlink( "SBUILD-FINISHED" ) if $main::batchmode;
-
-
-Process is as follows:
-
-1. unpack the chroot tarball (must unpack to chroot-autobuild)
-   Failure -> BUILDERFAIL
-2. mount up the proc, and devpts dirs
-   Failure -> BUILDERFAIL
-3. do apt-get update, apt-get dist-upgrade in the chroot
-   Failure -> CHROOTFAIL
-4. Copy into the dir the relevant dsc, orig (and diff if needed)
-5. sbuild --nolog --batch --distribution autobuild foo.dsc
-   Failure -> check build-progress
-              DEPFAIL / PACKAGEFAIL
-6. sudo ./kill-any-chrooted-processes.py
-   Failure -> BUILDERFAIL
-7. umount the proc, and devpts dirs
-   Failure -> BUILDERFAIL
-8. unlink the chroot (rm -rf or whatever as root)
-   Failure -> BUILDERFAIL
-9. OK
-
-We may make 8 an out-of-band process in which case it can't return BUILDERFAIL

=== removed file 'lib/canonical/buildd/uinotes.txt'
--- lib/canonical/buildd/uinotes.txt	2007-10-31 19:03:15 +0000
+++ lib/canonical/buildd/uinotes.txt	1970-01-01 00:00:00 +0000
@@ -1,68 +0,0 @@
-Launchpad Build UI notes
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-There are various parts to the buildd network which will have status
-information which can be presented to various of the users.
-
-The following classes of users exist...
-
- o Boggy standard user
- o Distribution build manager (member of some team or something)
- o Package maintainer (independant)
- o Package maintainer (in a distro context)
- o Distribution security team member (can see hidden builds)
- o Canonical buildd maintainer
-
-The buildd maintainer view
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The buildd maintainer wants to know if there are any builds failing
-due to buildd-related issues. I.E. if any builds are returning
-CHROOTFAIL or BUILDERFAIL
-
-The buildd maintainer also wants to see the build queues and the
-states of each buildd in the farm.
-
-The boggy-standard user view
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A plain and simple user (E.g. unauthenticated web access) can only see
-build logs of unrestricted packages. They can see if a package is
-queued to be built but no information on when it may be built.
-
-The independant package maintainer
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-An independant package maintainer will basically see the above and
-also an ETA for each architecture's package build for their packages.
-
-For failed builds they can obviously see build logs.
-
-The package maintainer within a distro
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A distro package maintainer gets to see the ETA of their package in
-each of the arch queues; also what packages are ahead of it.
-
-Distribution security team member
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Gets to see all the above; but also for restricted distroseries
-
-Distribution build manager
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Gets to see an overview of all the queues and can give back packages
-which managed to enter a manual dep-wait state or similar.
-
-
-Pages we're likely to need
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
- o A view for the buildd maintainer. How to designate this I am
-   unsure; perhaps this ought to be a program to run to generate the
-   info rather than a web page. [[TODO: Ask lamont about this]]
- o Build log viewer
- o Build queue viewer, optionally limited to a specific
-   build/sourcepackagerelease/sourcepackage/distroseries/arch
- o A way to give-back packages to re-try the build