← Back to team overview

maria-developers team mailing list archive

Updated (by Knielsen): Use Buildbot to populate apt/yum repositories (117)

 

-----------------------------------------------------------------------
                              WORKLOG TASK
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
TASK...........: Use Buildbot to populate apt/yum repositories
CREATION DATE..: Wed, 12 May 2010, 07:04
SUPERVISOR.....: Knielsen
IMPLEMENTOR....: Knielsen
COPIES TO......: 
CATEGORY.......: Other
TASK ID........: 117 (http://askmonty.org/worklog/?tid=117)
VERSION........: Server-9.x
STATUS.........: Assigned
PRIORITY.......: 60
WORKED HOURS...: 4
ESTIMATE.......: 16 (hours remain)
ORIG. ESTIMATE.: 20

PROGRESS NOTES:

-=-=(Knielsen - Wed, 12 May 2010, 18:25)=-=-
High-Level Specification modified.
--- /tmp/wklog.117.old.12634    2010-05-12 18:25:58.000000000 +0000
+++ /tmp/wklog.117.new.12634    2010-05-12 18:25:58.000000000 +0000
@@ -12,9 +12,35 @@
 This of course requires that it is possible to sign the packages after the
 actual build.
 
-For .rpm this seems to be easy (from reading, didn't try yet):
+----
+
+Here is how to sign the .rpms.
+
+Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros.
+
+Run
+
+    rpm --addsign *.rpm
+
+That's all! This can be tested by creating a local yum repository:
 
-    rpm --addsign <packages>
+    createrepo <dir>
+
+(where <dir> contains the signed .rpms). Then create the file
+/etc/yum.repos.d/localmaria.repo:
+
+[localmaria]
+name=Local MariaDB repo
+baseurl=file:///home/buildbot/rpms
+gpgcheck=1
+enabled=1
+gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg
+
+Now this should work to install MariaDB:
+
+    sudo yum install MariaDB-server
+
+----
 
 For .deb, I *think* we are using secure apt, which does not actually sign the
 packages, rather it signs the "Release" file which is created when the

-=-=(Knielsen - Wed, 12 May 2010, 07:14)=-=-
High-Level Specification modified.
--- /tmp/wklog.117.old.401      2010-05-12 07:14:27.000000000 +0000
+++ /tmp/wklog.117.new.401      2010-05-12 07:14:27.000000000 +0000
@@ -1 +1,35 @@
+As for signing, I think it may be possible/best to do the signing outside of
+buildbot, as a separate process. There are some advantages to this:
+
+ - Security: the private key can be kept less exposed when it is not included
+   in the buildbot infrastructure.
+
+ - It is good to have one step of human intervention before actually signing
+   and releasing packages.
+
+ - Generally reducing the complexity of the buildbot setup.
+
+This of course requires that it is possible to sign the packages after the
+actual build.
+
+For .rpm this seems to be easy (from reading, didn't try yet):
+
+    rpm --addsign <packages>
+
+For .deb, I *think* we are using secure apt, which does not actually sign the
+packages, rather it signs the "Release" file which is created when the
+repository is set up. So in this case again there is no problem doing the
+signing outside of the build itself (in fact that is the way it must be).
+
+Found two tools that can help with building and signing apt repositories:
+reprepro (seems to be the newest, recommended) and apt-ftparchive.
+
+----
+
+ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen
+suggested splitting up so we have this package ourselves, or maybe it can be
+handled with replace/provide/conflict dependencies.
+
+ToDo: Figure out exactly what files/directory structure needs to be uploaded
+(asked Peter, awaiting reply).
 

-=-=(Knielsen - Wed, 12 May 2010, 07:06)=-=-
Upgraded lucid VMs to the official release.

Discussed with Arjen how to handle things.

Did a lot of reading on how apt repositories work.

Worked 4 hours and estimate 16 hours remain (original estimate unchanged).



DESCRIPTION:

Since the package building for MariaDB is now fully automated in Buildbot, it
has been decided to use packages from Buildbot for the OurDelta apt and yum
repositories.

This worklog is about fixing/implementing anything that is missing to achieve
this.

 - When doing a real release build, packages/repositories need to be signed,
   so that users will not get a warning about unauthenticated packages. This
   signing must only be done on official releases, not on daily builds (to
   avoid confusing one with the other).

 - Packages must be uploaded from the Buildbot host. The OurDelta
   infrastructure has a DropBox share that could be used for this, another
   option is to simply use rsync.

 - Ubuntu 10.04 "lucid" has been released, and we need to support that for
   packages, so the Buildbot VM for lucid must be upgraded to have the
   official release.

 - In Ubuntu 10.04, the official MySQL packages include a new package
   mysql-client-core, we currently have a conflict with this on install that
   we need to handle somehow.


HIGH-LEVEL SPECIFICATION:



As for signing, I think it may be possible/best to do the signing outside of
buildbot, as a separate process. There are some advantages to this:

 - Security: the private key can be kept less exposed when it is not included
   in the buildbot infrastructure.

 - It is good to have one step of human intervention before actually signing
   and releasing packages.

 - Generally reducing the complexity of the buildbot setup.

This of course requires that it is possible to sign the packages after the
actual build.

----

Here is how to sign the .rpms.

Copy in the ourdelta/bakery signing stuff to ~/.gnupg and ~/.rpmmacros.

Run

    rpm --addsign *.rpm

That's all! This can be tested by creating a local yum repository:

    createrepo <dir>

(where <dir> contains the signed .rpms). Then create the file
/etc/yum.repos.d/localmaria.repo:

[localmaria]
name=Local MariaDB repo
baseurl=file:///home/buildbot/rpms
gpgcheck=1
enabled=1
gpgkey=http://master.ourdelta.org/deb/ourdelta.gpg

Now this should work to install MariaDB:

    sudo yum install MariaDB-server

----

For .deb, I *think* we are using secure apt, which does not actually sign the
packages, rather it signs the "Release" file which is created when the
repository is set up. So in this case again there is no problem doing the
signing outside of the build itself (in fact that is the way it must be).

Found two tools that can help with building and signing apt repositories:
reprepro (seems to be the newest, recommended) and apt-ftparchive.

----

ToDO: Figure out how to handle the mysql-client-core issue on lucid. Arjen
suggested splitting up so we have this package ourselves, or maybe it can be
handled with replace/provide/conflict dependencies.

ToDo: Figure out exactly what files/directory structure needs to be uploaded
(asked Peter, awaiting reply).


ESTIMATED WORK TIME

ESTIMATED COMPLETION DATE
-----------------------------------------------------------------------
WorkLog (v3.5.9)